(N 



O 



o 
in 



X 



Improved Bounds on the Parity-Check Density and 
Achievable Rates of Binary Linear Block Codes with 
Applications to LDPC Codes 



in 
o 

Gil Wiechman leal Sason 

j^V Technion - Israel Institute of Technology 

Haifa 32000, Israel 
{igillwOtx , sasonOee} . technion .ac.il 



February 1, 2008 



Abstract 



We derive bounds on the asymptotic density of parity-check matrices and the achievable rates of binary 
linear block codes transmitted over memoryless binary-input output-symmetric (MBIOS) channels. The 



> 

I/"} ■ lower bounds on the density of arbitrary parity-check matrices are expressed in terms of the gap between the 



rate of these codes for which reliable communication is achievable and the channel capacity, and the bounds 



are valid for every sequence of binary linear block codes. These bounds address the question, previously 

in ■ 

considered by Sason and Urbanke, of how sparse can parity-check matrices of binary linear block codes 
be as a function of the gap to capacity. Similarly to a previously reported bound by Sason and Urbanke, 

O ■ 

the new lower bounds on the parity-check density scale like the log of the inverse of the gap to capacity 



but their tightness is improved (except for a binary symmetric/erasure channel, where they coincide with 
the previous bound). The new upper bounds on the achievable rates of binary linear block codes tighten 
previously reported bounds by Burshtein et ai, and therefore enable to obtain tighter upper bounds on the 
thresholds of sequences of binary linear block codes under ML decoding. The bounds are applied to low- 
density parity-check (LDPC) codes, and the improvement in their tightness is exemplified numerically. The 
upper bounds on the achievable rates enable to assess the inherent loss in performance of various iterative 
decoding algorithms as compared to optimal ML decoding. The lower bounds on the asymptotic parity-check 
density are helpful in assessing the inherent tradeoff between the asymptotic performance of LDPC codes 
and their decoding complexity (per iteration) under message-passing decoding. 

Index Terms: Block codes, channel capacity, error probability, iterative decoding, linear codes, 
low-density parity-check (LDPC) codes, maximum-likelihood (ML) decoding, thresholds. 

1 Introduction 



Error correcting codes which employ iterative decoding algorithms are now considered state of the 
art in the field of low-complexity coding techniques. The graphical representation of these codes is 



used to describe their algebraic structure, and also enables a unified description of their iterative 
decoding algorithms over various channels. These codes closely approach the capacity limit of many 
standard communication channels under iterative decoding. By now, there is a large collection of 
families of iteratively decoded codes including low-density parity-check (LDPC), turbo, repeat- 
accumulate and product codes; all of them, demonstrate a rather small gap (in rate) to capacity 
with feasible complexity. In [6], Khandekar and McEliece have suggested to study the encoding 
and decoding complexities of ensembles of iteratively decoded codes on graphs as a function of 
their gap to capacity. They conjectured that if the achievable rate under iterative message-passing 
decoding is a fraction 1 — e of the channel capacity, then for a wide class of channels, the encoding 
complexity scales like In ^ and the decoding complexity scales like ^ In ^ . The only exception is the 
binary erasure channel (BEC) where the decoding complexity behaves like In i (same as encoding 
complexity) because of the absolute reliability of the messages passed through the edges of the 
graph (hence, every edge can be used only once during the iterative decoding process). 

LDPC codes are efficiently encoded and decoded due to the sparseness of their parity-check 
matrices. In his thesis [4], Gallager proved that right-regular LDPC codes (i.e., LDPC codes with a 
constant degree (or) of the parity-check nodes) cannot achieve the channel capacity on a BSC, even 
under optimal ML decoding. This inherent gap to capacity is well approximated by an expression 
which decreases to zero exponentially fast in or. Richardson et al. [11] have extended this result, 
and proved that the same conclusion holds if or designates the maximal right degree. Sason and 
Urbanke later observed in [13] that the result still applies when considering the average right degree. 
Gallager's bound [4, Theorem 3.3] provides an upper bound on the rate of right-regular LDPC codes 
which achieve reliable communications over the BSC. Burshtein et al. have generalized Gallager's 
bound for a general MBIOS channel [1], and the work in [13] relies on their generalization. 

Consider the number of ones in a parity-check matrix which represents a binary linear code, 
and normalize it per information bit (i.e., with respect to the dimension of the code). This quantity 
(which will be later defined as the density of the parity-check matrix) is equal to ^-j^ times the 
average right degree of the bipartite graph that represents the code, where R is the rate of the code 
in bits per channel use. In [13], Sason and Urbanke considered how sparse can parity-check matrices 
of binary linear block codes be, as a function of their gap to capacity (where this gap depends in 
general on the channel and on the decoding algorithm). An information-theoretic lower bound 
on the asymptotic density of parity-check matrices was derived in [13, Theorem 2.1] where this 
bound applies to every MBIOS channel and every sequence of binary linear block codes achieving a 
fraction 1 — e of the channel capacity with vanishing bit error probability. It holds for an arbitrary 
representation of parity-check matrices for these codes, and is of the form Kl+ ^J n e w h ere j{ l 
and K2 are constants which only depend on the channel. Though the logarithmic behavior of 
this lower bound is in essence correct (due to a logarithmic behavior of the upper bound on the 
asymptotic parity-check density in [13, Theorem 2.2]), the lower bound in [13, Theorem 2.1] is not 
tight (with the exception of the BEC, as demonstrated in [13, Theorem 2.3], and possibly also the 
binary symmetric channel (BSC)). The derivation of the bounds in this paper was motivated by 
the desire to improve the results in [1, Theorems 1 and 2] and [13, Theorem 2.1] which are based 
on a two-level quantization of the log- likelihood ratio (LLR). 

In [7], Measson and Urbanke derived an upper bound on the maximum-likelihood (ML) thresh- 
olds of LDPC ensembles when the codes are transmitted over the BEC. Their general approach 
relies on EXIT-like functions and the area theorem. This bound coincides with the ML threshold 
determined by Montanari et al. using the replica method, showing that the bound is in fact tight. 
In [8], Montanari presented a new approach for the analysis of codes on graphs under maximum 
a posteriori probability (MAP) decoding. His approach is based on statistical mechanics, and the 
resulting expressions are related to the density evolution analysis of belief propagation decoding. 



Motivated by the heuristic statistical mechanics results, it was conjectured in [8] that the bounds 
on the asymptotic achievable rates of LDPC codes are tight. 

We derive in this paper improved bounds on the achievable rates and the asymptotic parity- 
check density of sequences of binary linear block codes. The bounds in [1, 13] and this paper 
are valid for every sequence of binary linear block codes, in contrast to a high probability result 
which was previously derived for the binary erasure channel (BEC) from density evolution analysis 
[14]. Shokrollahi proved in [14] that when the codes are communicated over a BEC, the growth 
rate of the average right degree (i.e., the average degree of the parity-check nodes in a bipartite 
Tanner graph) is at least logarithmic in terms of the gap to capacity. The statement in [14] is a 
high probability result, and hence it is not necessarily satisfied for every particular code from this 
ensemble. Further, it assumes a sub-optimal (iterative) decoding algorithm, where the statements 
in [1, 13] and this paper are valid even under optimal ML decoding. 

The significance of the bounds in this paper is demonstrated in two respects. The new upper 
bounds on the achievable rates of binary linear block codes tighten previously reported bounds 
by Burshtein et al. [1], and therefore enable to obtain tighter upper bounds on the thresholds of 
sequences of binary linear block codes under ML decoding. They are applied to LDPC codes, and 
the improvement in their tightness is exemplified numerically. Comparing the new upper bounds 
on the achievable rates with thresholds provided by a density-evolution analysis gives rigorous 
bounds on the inherent loss in performance due to the sub-optimality of iterative message-passing 
decoding (as compared to soft-decision ML decoding). The new lower bounds on the asymptotic 
parity-check density tighten the lower bound in [13, Theorem 2.1]. Since the parity-check density 
can be interpreted as the complexity per iteration under iterative message-passing decoding, then 
tightening the reported lower bound on the parity-check density [13] gives insight on the tradeoff 
between the asymptotic performance and decoding complexity of LDPC codes. 

In this paper, preliminary material is presented in Section 2, and the theorems are introduced 
and proved in Sections 3 and 4. The derivation of the bounds in Section 3 was motivated by 
the desire to generalize the results in [1, Theorems 1 and 2] and [13, Theorem 2.1]. A two-level 
quantization of the log- likelihood ratio (LLR), in essence replacing the arbitrary MBIOS channel 
by a physically degraded binary symmetric channel (BSC), is modified in Section 3 to a quantized 
channel which better reflects the statistics of the original channel (though the quantized channel 
is still physically degraded w.r.t. the original channel). The number of quantization levels of the 
LLR for the new channel is an arbitrary integer power of 2, and the calculation of these bounds is 
subject to an optimization of the quantization levels, as to obtain the tightest bounds within their 
form. In Section 4, we rely on the conditional pdf of the LLR of the MBIOS channel, and operate 
on an equivalent channel without quantizing the LLR. This second approach finally leads to bounds 
which are uniformly tighter than the bounds we derive in Section 3. It appears to be even simpler 
to calculate the un-quantized bounds in Section 4, as their calculation do not involve the solution 
of any optimization equation (in contrast to the quantized bounds, whose calculation involves a 
numerical solution of optimization equations w.r.t. the quantization levels of the LLR). Fortunately, 
the multi-dimensional integral obtained in the derivation of the bounds in Section 4 is transformed 
to a rapidly convergent infinite series of one-dimensional integrals; this issue is crucial in facilitating 
the calculation of the bounds in Section 4. We note that the significance of both the quantized 
and un-quantized bounds in Sections 3 and 4, respectively, stems from a comparison between these 
bounds which gives insight on the effect of the number of quantization levels of the LLR (even if 
they are optimally determined) on the achievable rates, as compared to the ideal case where no 
quantization is done. Numerical results are exemplified and explained in Section 5. Finally, we 
summarize our discussion in Section 6 and present interesting issues which deserve further research. 
Four appendices provide further technical details referring to the proofs in Sections 3 and 4. 



2 Preliminaries 



We introduce here some definitions and theorems from [1, 13] which serve as a preliminary material 
for the rest of the paper. Definitions 2.1 and 2.2 are taken from [13, Section 2]. 

Definition 2.1 (Capacity- Approaching Codes). Let {C m } be a sequence of codes of rate R m , 
and assume that for every m, the codewords of the code C m are transmitted with equal probability 
over a channel whose capacity is C. This sequence is said to achieve a fraction 1 — e of the channel 
capacity with vanishing bit (block) error probability if ]im m —^ 00 R m = (1 — e)C, and if there exists a 
decoding algorithm under which the average bit (block) error probability of the code C m tends to 
zero in the limit where m — > oo. 

Definition 2.2 (Parity-Check Density). Let C be a binary linear code of rate R and block 
length n, which is represented by a parity-check matrix H. We define the density of H, call it 
A = A(H), as the normalized number of ones in H per information bit. The total number of ones 
in H is therefore equal to nRA. 

Definition 2.3 (Log-Likelihood Ratio (LLR)). Let Py\x{'\') be the conditional pdf of an 
arbitrary MBIOS channel. The log- likelihood ratio (LLR) at the output of the channel is 

w Wr(y|* = i)/ 



Throughout the paper, we assume that all the codewords of a binary linear block code are 
equally likely to be transmitted. Also, we use the notation h,2{ ) for the binary entropy function to 
base 2, i.e., h 2 {-) = — xlog 2 (o;) — (1 — x) log 2 (l — x). 

Theorem 2.1 (An Upper Bound on the Achievable Rates for Reliable Communication 
over MBIOS Channels). [1, Theorem 2]: Consider a sequence {C m } of binary linear block codes 
of rate R m , and assume that their block length tends to infinity as m — > oo. Let H m be a parity- 
check matrix of the code C m , and assume that d\.^ m designates the fraction of the parity-check 
equations involving k variables. Let 



eZfc = liminf m , R = liminf R m . (1) 

m^oo 



Suppose that the transmission of these codes takes place over an MBIOS channel with capacity C 
bits per channel use, and let 
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W= 2 



I poo 

- J mm{f(y),f(-y)) dy (2) 

where f(y) = Py\x{u\X = 1) designates the conditional pdf of the output of the MBIOS channel. 
Then, a necessary condition for vanishing block error probability as m — > oo is 

*<i— 1 " c 
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Theorem 2.2 (Lower Bounds on the Asymptotic Parity-Check Density with 2-Levels 
Quantization). [13, Theorem 2.1]: Let {C m } be a sequence of binary linear codes achieving a 
fraction 1 — e of the capacity of an MBIOS channel with vanishing bit error probability. Then, the 
asymptotic density (A m ) of their parity-check matrices satisfies 

liminfA m >^- £ - (3) 



where 



K x = 



(1-C) ln( 




) 



K 2 = 



1-C 



(4) 
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and w is denned in (2). For a BEC with erasure probability p, the coefficients K\ and if 2 in (4) 
are improved to 



Using standard notation, an ensemble of (n, A, p) LDPC codes is characterized by its length n, 
and the polynomials A(x) = ^i x% l an d P( x ) = ^£2 z 9 * 2 ^ -1 ' where Aj (pj) is equal to the 

probability that a randomly chosen edge is connected to a variable (parity-check) node of degree i. 
The variables (parity-check sets) are represented by the left (right) nodes of a bipartite graph which 
represents an LDPC code. 

3 Approach I: Bounds Based on Quantization of the LLR 

In this section, we introduce bounds on the achievable rates and the asymptotic parity-check density 
of sequences of binary linear block codes. The bounds generalize previously reported results in [1] 
and [13] which were based on a symmetric two-level quantization of the LLR. This is achieved by 
extending the concept of quantization to an arbitrary integer power of 2; to this end, our analysis 
relies on the Galois field GF(2 m ). In Section 3.1, we demonstrate the results and their proofs for 
four-level quantization. In Section 3.2, we extend the results to a symmetric quantization with a 
number of levels which is an arbitrary integer power of 2. This order of presentation was chosen since 
many concepts which are helpful for the generalization in Section 3.2 are written in a simplified 
notation for the four-level quantization, along with all the relevant lemmas for the general case 
which are already introduced in the derivation of the bound with four-level quantization. This also 
shortens considerably the proof for the general quantization in Section 3.2. 

3.1 Bounds for Four-Levels of Quantization 

As a preparatory step towards developing bounds on the parity-check density and the rate of binary 
linear block codes, we present a lower bound on the conditional entropy of a transmitted codeword 
given the received sequence at the output of an arbitrary MBIOS channel. 

Proposition 3.1. Let C be a binary linear block code of length n and rate R. Let x = (xi, . . . x n ) 
and y = (yi, . . . , y n ) designate the transmitted codeword and received sequence, respectively, when 
the communication takes place over an MBIOS channel with conditional pdf py\ x{'\')- F° r an 
arbitrary positive I € M + , let us define the probabilities Po,Pi,P2,P3 as follows: 




(5) 



po = Pr{LLR(Y) > I | X = 0} 

pi = Pr{LLR(Y) G (0, 1] \ X = 0} + - Pr{LLR(Y) = [ X = 0} 

p 2 = Pr{LLR(Y) G [-/, 0) | X = 0} + i Pr{LLR(Y) = | X = 0} 
p 3 ± p r {LLR(Y) < -I I X = 0}. 



(6) 



For an arbitrary parity-check matrix of the code C, let dk designate the fraction of the parity-checks 
involving k variables. Then, the conditional entropy of the transmitted codeword given the received 
sequence satisfies 



"< X ' Y >>1-C-(1-A) 



n 




Proof. Considering an MBIOS channel whose conditional pdf is given by Py\x('\')i we introduce a 
new physically degraded channel. It is a binary- input, quaternary-output symmetric channel (see 
Fig. 1). To this end, let I 6 M + be an arbitrary positive number, and let a be a primitive element of 
the Galois field GF(2 2 ) (so a 2 = 1 + a). The set of the elements of this field is {0, 1, a, 1 + a}. Let 
Xi and Yi designate the random variables referring to the input and output of the original channel 
Py\x('\') & t time i (where i = 1, 2, . . . , n). We define the degraded channel as a channel with four 
quantization levels of the LLR. The output of the degraded channel at time i, Zi, is calculated from 
the output yi of the original channel as follows: 

If LLR(yj) > I, then Zi = 0. 

If < LLR(yj) < /, then Zi = a. 

If -/ < LLR(yi) < 0, then z { = 1 + a. 

If LLR(yj) < —I, then Zj = 1. 

If LLR(y«) = 0, then z\ is chosen as a or 1 + a with equal probability (|). 




Figure 1: The channel model in the left plot is a physically degraded channel used for the derivation 
of the bound with four levels of quantization. The element a denotes a primitive element in GF(2 2 ). 
This channel model is equivalent to a channel with an additive noise in GF(2 2 ) (see right plot). 

From the definition of the degraded channel in Fig. 1, this channel has an additive noise and is 
also binary-input output-symmetric. It follows that the transition probabilities given in (6) can be 
expressed in an equivalent way by 

Po = Pr(Z = I X = 0) = Pr(Z = 1 | X = 1) 
Pl = p r (Z = a I X = 0) = Pr(Z = 1 + a | X = 1) 
p 2 = p r (Z = 1 + a I X = 0) = Pr(Z = a \ X = 1) 
p 3 = p r (Z = 1 I X = 0) = Pr(Z = | X = 1) 



where the symmetry in these transition probabilities holds since the original channel is MBIOS. 

Since C is a binary linear block code of length n and rate R, and the codewords are transmitted 
with equal probability then 

H(X) = nR. (8) 
Also, since the channel Py\x('\') is memoryless, then 

H(Y\X) = nH(Y\X). (9) 



We designate the output sequences of the original channel and its degraded version by Y and Z, 
respectively. Since the mapping from Yj to the degraded output Z^ (i = 1, 2, • ■ ■ , n) is memoryless, 
then H(Z\Y) = nH(Z\Y), and 

H(Y) = H(Z) - H(Z\Y) + H(Y\Z) 

= H(Z) -nH(Z\Y) + H(Y\Z) (10) 



H(Y\Z) < J2 H ( Y i\ Z i) 

i=i 

= nH(Y\Z) 

= n[H(Y)-H(Z) + H(Z\Y)]. (11) 

Applying the above towards a lower bound on the conditional entropy H(K\Y), we get 

H(X\Y) = H(X) + #(Y|X) - H(Y) 

= nR + nH(Y\X) - H(Y) 

= nR + nH(Y\X) - H{Z) - H{Y\Z) +nH(Z\Y) 

> nR + nH(Y\X) - H{Z) - n [H(Y) - H(Z) + H(Z\Y)\ + nH(Z\Y) 
= nR - H{Z) + riH(Z) - n [H(Y) — H(Y\X)] 

= nR- H(Z) +nH{Z) -nI{X;Y) 

> nR- H{Z)+nH{Z)-nC (12) 

where the second equality relies on (8) and (9), the third equality relies on (10), the first inequality 
relies on (11), and I(X;Y) < C is used for the last transition (where C designates the capacity 
of the non-degraded channel). In order to obtain a lower bound on iT(X|Y) from (12), we will 
calculate the exact entropy of the random variable Z, and find an upper bound on the entropy of 
the random vector Z. This will finally provide the lower bound in (7). 

Since C is a binary linear block code, then the input X is equally likely to be zero or one. The 
output Z of the degraded channel in Fig. 1 has the following probability law: 

Pr(Z = 0) = Pr(Z = 0, X = 0) + Pr(Z = 0, X = 1) 

= Pr(Z = | X = 0) Pr(A = 0) + Pt(Z = | X = 1) Pt(X = 1) 
Po +P3 
2 

and in a similar manner, 

D ,y -i \ P0+P3 D f ry s. D fry , , \ Pi + P2 

Pr(z = 1) = — - — , Pr(z = a) = Pr(z = 1 + a) = — - — . 



The entropy of the random variable Z is therefore equal to 



H ( z) _ 2 (sale) 1* (-?-) +2 (et±is) k*(-?-) 

V 2 / V 2 / \Pl+P2j 

= 1 + (po + p 3 )log 2 ( 1 J + (pi +p 2 )log 2 ( ) 

= l + / i2 (pi+p 2 ) (13) 

where the last transition follows from the equality po + pi +P2 +Pz = 1- We now derive an upper 
bound on the entropy H(Z). To this end, let 

Z i = @ i + <f> i a, i = l,2,...,n (14) 

where = (61,62, . . . , 6„) and 3> = ($1, $2, . . . , 3> n ) are random vectors over {0, 1}™. From the 
composition of into a pair of two binary components (6j, $j), it follows from Fig. 1 that 

Pr($i = 0) =pi +p 2 , Pr($i = 1) =p +P3 = 1 - (Pi +P2), i=l,2,...,n. (15) 

Based on (14) and (15), it is easy to verify the following chain of equalities: 

H(Z) = H(Z 1 ,Z 2 ,...,Z n ) 

= ir(ei,e 2 ,...,e n ,$i,$ 2 ,...,*n) 

= fl-(* 1 ,* 2 ,...,* n ) + fr(e 1 ,e 2 ,...,e n | * 1 ,* 2 ,...,* n ) 

= n^ 2 (pi+p 2 )+i7(6i,6 2 ,...,6 n I $i,$ 2 ,...,$ n ) (16) 

where the last equality follows from (15) and since the degraded channel in Fig. 1 is memoryless. 
Let us define the syndrome at the output of the degraded channel as 

s4(e ll e 2l ...,e n ) h t 

where H is a parity-check matrix of the binary linear block code C. We note that the calculation 
of the syndrome only takes into account the first components in the composition of the vector Z 
in (14). Note that the transmitted codeword x G C only affects the 0-components of the vector 
Z in (14), and also xi7 T = for any such a codeword. Let us define L as the index of the vector 
(61, 2 , . . . , n ) in the coset referring to the syndrome S. Since each coset has exactly 2 nR elements 
which are equally likely, then H(L) = nR, and 

#(01,02,..., 0„ I $i,$ 2 ,...,$ n ) = H(S,L I $l,$ 2 , • • • ,$n) 

< H(L) + H(S I $i,$ 2 , ■•• ,$n) 

= nR + H(S I $i,$ 2 ,...,$ n ). (17) 

Considering a parity-check equation involving k variables, let {11,12, ■ ■ ■ iik} be the set of indices 
of the variables involved in this parity-check equation. The relevant component of the syndrome S 
which refers to this parity-check equation is equal to zero or one if and only if the components of 
the sub- vector (0^ , 0j 2 , . . . , 0j fe ) differ from the components of the sub- vector (X^ , Xi 2 , . . . , Xi k ) 
in an even or odd number of indices, respectively. It is clear from Fig. 1 that for an index i for 
which 3>j = 1, the random variables and 0j are different in probability ^ ; as a result of the 
symmetry of the channel, this probability is independent of the value of X{. Similarly, for an index 
i for which <3?j = 0, the random variables and 0^ are different in probability p( ^ p:i , which again 
is independent of the value of Xj. 



Given that the Hamming weight of the vector $i 2 , • • • , &i k ) is t, then the probability that 
the components of the two random vectors (0^ , 0j 2 , . . . , @i k ) and (X^ , Xj 2 , . . . , Xi k ) differ an even 
number of times is equal to 

ft (t, k) q 2 (t, k) + (1 - ?1 (t, fc)) (1 - q 2 {t, k)) 

where qi(t,k) designates the probability that among the t indices i for which $j = 1, the random 
variables Xi and 0, differ an even number of times, and q 2 (t,k) designates the probability that 
among the k — t indices i for which <l?j = 0, the random variables Xi and 0j differ an even number 
of times. Based on the discussion above, it follows that 

t 



t m rem * ' * 



P2 \ V P1+P2 



i 1 + ( 1 2^ 



P1+P2J 2 

N , .. . I + [ 1 - -2£3_V _ * 

k — t\ ( p 3 \ ^ \ P0+P3 J 



B(( , t) _ E (*T*)(_e_y_^l! 

■ V * / VP0+P3/ 
i even 



so the probability that the two vectors (0^ , 0j 2 , . . . , &i k ) and (X^ , Xj 2 , . . . , X; lk ) differ in an even 
number of indices is 



1.+ ( 1 - -^M ' 

qi (t, k) q 2 (t, k) + (1 - gi(t, fc)) (1 - g 2 (i, fc)) = 



r V P1+P2 J v P0+P3 ; 



2 

We conclude that given a vector $ £ {0, l} fe of Hamming weight t 



H(Si I ($ il ,$ i2 ,...,$ ifc ) = $) = /* 2 



V P1+P2 / V P0+P3 / 



V / 

This yields that if the calculation of a component Si (i = 1, . . . , n(l — i?)) in the syndrome S relies 
on a parity-check equation involving variables, then 

H(Si I $i,$ 2 , • • • ,$n) = I $^,^2,...,^) 

= E Pr >*«>••• >***) = *) I ■■■>$i fc ) =*) 

$e{o,i} fc 



E/ fc \/ \t/ \ML V Pl+P2^ V P0+P 3 / 
^J(P1+P2)*(P0+P3)' C % " " 



t=0 

k 



^2( k ^)(Pl+P2) t (po+P3) k l h 2 




t=0 

where the third equality turns to averaging over the Hamming weight of $ = , <3?j 2 , . . . , <3?i fe ) , and 
the last equality follows from the symmetry of the binary entropy function (where h 2 (x) = h 2 (l — x) 
for x € [0, 1]). Let dk designate the fraction of parity-check equations in the arbitrary parity-check 
matrix which involve k variables, so their total number is nil — R)dk and 

H(S I $!,$ 2 ,...,$ n ) 

n(l-R) 

< £ H(Si I $!,$ 2 ,...,$ n ) 



1=1 



d-«)Ej*E (t) ^ + »)'(P» + W)*"^ f 1 (1 Pl2 " 2p2 { (l ^ )fct ) I -(18) 



By combining (16)-(18), an upper bound on the entropy of the random vector Z follows: 
H(Z) < nR + nh 2 (pi + p 2 ) 

1 _ Cl _ 2 P2 \t(i _ _2p 2 _\k" t \ " 
V P1+P2/ V P0+P3/ 1 



+n(l-R)Y J < 



t=o v/ 



V .(19) 



The substitution of (13) and (19) in (12) finally provides the lower bound on the conditional entropy 
H(X | Y) in (7). □ 

The following theorem tightens the lower bound on the parity-check density of an arbitrary 
sequence of binary linear block codes given in [13, Theorem 2.1]: 

Theorem 3.1 ("Four-Level Quantization" Lower Bound on the Asymptotic Parity-Check 
Density of Binary Linear Block Codes). Let {C m } be a sequence of binary linear block codes 
achieving a fraction 1 — e of the capacity of an MBIOS channel with vanishing bit error probability. 
Then, the asymptotic density (A m ) of their parity-check matrices satisfies 

Ki + K 2 In ± 

liminf A m > — e - (20) 

m^oo 1 — e 



where 



Kl = K > 111 Gh® Hr) • K ' = - cln( ijJ^ C + ^5E) <21> 

V (P1+P2) P0+P3 / 

and P0iPiiP2,P3 are defined in (6) in terms of I G M + . The optimal value of I is given implicitly 
by the equation 

v\ + e~ l pj = pi + e-'pg 
{P1+P2) 2 (P0+P3) 2 

where such a solution always exists. 1 

Proof. Derivation of the lower bound in (20) and (21): 

Lemma 3.1. Let C be a binary linear block code of length n and rate R. Let P^ designate 
the average bit error probability of the code C which is associated with an arbitrary decoding 
algorithm and channel, and let X and Y designate the transmitted codeword and received sequence, 
respectively. Then 

H(X I Y) 



n 



<Rh 2 (P h ). (23) 



Proof. The lemma is proved in Appendix A.l. □ 
Lemma 3.2. h 2 (x) < 1 - ^ {\ - x) 2 for < x < 1. 

Proof. The lemma is proved in [13, Lemma 3.1]. □ 

x It was observed numerically that the solution / of the optimization equation (22) is unique when considering the 
binary-input AWGN channel. We conjecture that the uniqueness of such a solution is a property which holds for 
MBIOS channels under some mild conditions. 



Referring to an arbitrary sequence of binary linear block codes {C m } which achieves a fraction 
1 — e to capacity with vanishing bit error probability, then according to Definition 2.1, there exists 
a decoding algorithm (e.g., ML decoding) so that the average bit error probability of the code C m 
tends to zero as m goes to infinity, and linim^oo R m = (1 — e)C. From Lemma 3.1, we obtain that 
linim^oo — - — z~ — — = where X m and Y m designate the transmitted codeword in the code C m 
and the received sequence, respectively , and n m designates the block length of the code C m . From 
Proposition 3.1, we obtain 



H(X. m \Y ri 



n r , 



> 1 - C - (1 - Rm) 



•E 



t=0 



2p2 \ f 

P1+P2 



Yd _ 2 P3 

I V P0+P3' 



> . 



By letting m go to infinity, then 



1 _ C - (1 - (1 - e)C) 1 4 E (J) (Pi + (Po + P3) fe ~% 

and the upper bound on ^(O i n Lemma 3.2 gives 
1-C- (l-(l-e)C) • 



I _ ( PI ~P2 Y( P0~P3 ^ fc-t 



< 



•Ei^Ei J(pi+P2)*(po+p 3 ) 



k-t 



1 



Pi -P2 



2 In 2 Vpi + P2 



2f 



PO ~P3 \ 
P0+P3J 



2{k-t) 



< 0. (24) 



Since Po + Pi + P2 + P3 = 1 (i-e., the transition probabilities of the channel in Fig. 1 sum to 1), 
then 



t=o 



Ei^Ew (pi+P2) i (p +p 3 ) fc 



1 



Pi -P2 



21 



E * 



1 - 
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t=o 
fc 



t=0 



2 In 2 \pi + p 2 

(pi -P2) 2 V / (PO -P3) 2 \ fc ~* 
Pi + P2 / V PO + P3 / 

(PO -P3j 



PO ~ P3 \ ^ ' 
P0+P3J 



k\ ( (pi - P2) 



2\ * 



Pi +P2 



< 1 



2 In 
1 



— yUk 

n2 ^ I 
k \ 



(pi -P2) 2 , (po -p 3 ) 



Pi +P2 



PO +P3 
2\ fc 1 



+ 



(Pi -P2) 2 , (PO -P3 



PO +P3 



2\ a R 



2 In 2 V P1+P2 



+ 



Po +P3 



(25) 



where ap, = kd^ designates the asymptotic average right degree of the bipartite graphs which 
refer to the sequence of linear block codes {C m }, and the last transition follows from Jensen's 
inequality. Combining (24) and (25) gives 



1-C - (1 - (1 -e)C) 



1 - 



2 In 2 



(Pi -P2) 2 (Po -psf 

Pi + P2 PO + P3 



2 \ a Rl 



< 0. 



This yields the following lower bound on the asymptotic average right degree: 

a R >K[ + K^hi Q 



(26) 



where 



K ' = m V21n2 C I K ' = i (of) 

1 ln Api-p 2 ) 2 , (po-p 3 ) 2 \ ' 2 ln / (Pi- P2 ) 2 , (Po-p 3 ) a y v ; 

\ P1+P2 P0+P3 / V P1+P2 P0+P3 / 



According to Definition 2.2, the density (A) of a parity-check matrix is equal to the number of 
edges in the corresponding bipartite graph normalized per information bit, while the average right 
degree (or) is equal to the same number of edges normalized per parity-check node. These different 
scalings of the number of the edges in a bipartite graph therefore imply that 

A = ^ a R (28) 

where R is the rate of a binary linear block code. By our assumption, the asymptotic rate of the 
sequence of code {C m } is equal to a fraction 1 — s of the capacity. Therefore, by combining (26) and 
(28) with R = (1 — e)C, we obtain a lower bound on the asymptotic parity-check density which is 
of the form 

K 1 +K 2 \n{\) 
l-e 

where 

Ki,2= l -^-K[ t2 (29) 

and K[ 2 are introduced in (27). This completes the proof of the lower bound in (20) with the 
coefficients K\p, in (21). 

Derivation of the optimization equation (22): We refer the reader to Appendix A. 2, where we 
also show the existence of such a solution. □ 



Discussion: It is required to show that we achieve an improved lower bound on the parity-check 
density, as compared to the one in [13, Theorem 2.1]. To this end, it suffices to show that 

{P1 ~ P2)2 + ^-P^ > { l-2wf 

Pi +P2 P0+ P3 

For a proof of this inequality, we refer the reader to Appendix A. 3. 

This therefore proves that the new lower bound is tighter (i.e., larger) than the original bound 
in [13, Theorem 2.1] (which corresponds to a two-level quantization of the LLR, as compared to 
the new bound which is based on a four-level quantization of the LLR). 

Based on the proof of Theorem 3.1, we prove and discuss an upper bound on the asymptotic 
rate of every sequence of binary linear codes for which reliable communication is achievable. The 
bound refers to soft-decision ML decoding, and it is therefore valid for any suboptimal decoding 
algorithm. Hence, the following result also provides an upper bound on the achievable rate of 
ensembles of LDPC codes under iterative decoding where the transmission takes places over an 
MBIOS channel. The following bound improves the bounds stated in [1, Theorems 1 and 2]: 

Corollary 3.1 ("Four-Level Quantization" Upper Bound on the Asymptotic Achievable 
Rates of Sequences of Binary Linear Block Codes). Let {C m } be a sequence of binary linear 
block codes whose codewords are transmitted with equal probability over an MBIOS channel, and 
suppose that the block length of this sequences of codes tends to infinity asnn oo. Let dk^ m be 
the fraction of the parity-check nodes of degree k in an arbitrary representation of the code C m 



(30) 



by a bipartite graph. 2 Then a necessary condition for this sequence to achieve vanishing bit error 
probability as m — > oo is that the asymptotic rate R of this sequence satisfies 



R < 1 — max 



1-C 



2p3 \fc-t 



d*E ( t ) + ») 4 (P" + P^~ th > ( Pl+P2 2 P " +P 

2(P2+P 3 ) ] 



l~Y.dk (l-2(p2+p 3 ))' 



(31) 



where Po,Pi,P2,P3 are introduced in (6), and c4 and i? are introduced in (1). 



Proof. The first term in the RHS of (31) follows from (7) in Proposition 3.1 and (23) in Lemma 3.1. 
It follows directly by combining both inequalities, and letting the bit error probability Pb go to 
zero. The second term in the RHS of (31) follows from the proof of [13, Corollary 3.1] which is 
based on the erasure decomposition Lemma [11]. □ 



Considering ensembles of LDPC codes, we note that the fraction d^ of nodes of degree k is 
calculated in terms of the degree distribution p(-) by the equation 

pk 

4 = ---- • ( 32 ) 

p(x) dx 



o 



3.2 Extension of the Bounds to 2 d Quantization Levels 

Following the method introduced in Section 3.1, we commence by deriving a lower bound on the 
conditional entropy of a transmitted codeword given the received sequence. 

Proposition 3.2. Let C be a binary linear block code of length n and rate R. Let x = (xi, . . . x n ) 
and y = (y±, . . . , y n ) designate the transmitted codeword and received sequence, respectively, when 
the communication takes place over an MBIOS channel with conditional pdf Py\x{'\')- F° r an 
arbitrary d > 2 and < l 2 d-i_i < ■ . . < h < h < — °°, let us define the set of probabilities 
{Ps^Jq 1 as follows: 

' Pr{/ S+1 < LLR(y) < l 3 | X = 0} s 
Pr{0 < LLR(Y) < Z 2 d-i_! | X = 0} + \ Pr{LLR(F) = | X = 0} s 
Ps ~ I Prj-^d-i.! < LLR(y) < | X = 0} + \ Pr{LLR(y) = | X = 0} s 
, Pr{-/ 2d _ (s+1) < LLR(1") < -l 2d _ s X = 0} s 

For an arbitrary parity-check matrix of the code C, let dk designate the fraction of the parity-checks 
involving k variables. Then, the conditional entropy of the transmitted codeword given the received 

2 For a sequence of ensembles of binary linear codes {C m }, we denote by dk, m the probability of picking (with 
uniform distribution) a parity-check node of degree k from a bipartite graph which represents the code C m . 



= 0, ...,2 d - 1 -2 
= 2 d ~ 1 - 1 
= 2 d - 1 

= 2 d ~ 1 + l,...,2 d -l. 



(33) 



sequence satisfies 
H(X\Y 



n 



2 d -!-l / 

n (Pi+P2--i-i) fc< ^ ~ 



i=0 



n 1 



i=0 



1-i 



Pi +P 2 rf-l-j 



(34) 



Proof. Following the proof of Proposition 3.1, we introduce a new physically degraded channel. It 
is a memoryless binary- input 2 d -output symmetric channel (see Fig. 1 for d = 2). To this end, let 
h > ^2 > • • • > ^2 d - 1 -i € M + be arbitrary non-negative numbers, and denote Zo — °°- The output 
alphabet of the degraded channel is defined to be GF(2 d ) whose elements form the set 

'd-l 

a j s.t (a ,ai, . . . ,a d -i) € {0, 1} C 

j=0 



For s = 0, 1, . . . , 2 d l — 1 , let us denote the d— 1-bit binary representation of s by (a^, • • • , 



i.e. 



d-l 
3=1 

Let Xi and Y$ designate the random variables referring to the input and output of the original 
channel Py\x{'\') a t time i (where z = 1,2,..., n). As a natural generalization of the channel model 
in Fig. 1, we introduce a channel with 2 d quantization levels of the LLR. The output of the degraded 
channel at time i, Zi, is calculated from the output yi of the original channel as follows: 

If l s+1 < LLR(yi) < l s for some < s < 2 d ~ 1 - 1, then Zi = Y0jz{ a< f ] « j - 

If < LLR(yj) < / 2 <f-i_i, then zi = YljZi . 

If -Z 2 d-i_i < LLR(yj) < 0, then Zi = 1 + ^=1 a*. 

If -Z s < LLR(yj) < -/ s+ i for some < s < 2 d ~ x - 1, then z { = 1 + a J '. 

If LLR(yj) = 0, then Zj is chosen as Ylj=i aJ or 1 + Ylj=i °^ W1 th equal probability (i). 

From (33) , the transition probabilities of the degraded channel are expressed in an equivalent way 
by 

d-l d-l 

Ps = p r (Z = ^ a j \X = 0) = Pr(Z = 1 + ^ a j \ X = 1) 
3=1 i=i 

d— 1 d— 1 

p 2d _ 1 _ s = p r (Z = 1 + «j S) a J ' I X = 0) = Pr(Z = ^ a j \ X = 1) (35) 

3 1 ' j"=i 



where s = 0, 1, . . . , 2 d 1 — 1. The symmetry in these equalities holds since the channel is MBIOS. 



Equations (8)-(12) hold also for the case of 2 rf -level quantization. Thus, we will calculate the 
entropy of the random variable Z, and an upper bound on the entropy of the random vector Z. 
This will finally provide the lower bound in (34). 

Since X is equally likely to be zero or one, the output of the degraded channel, which is 
symmetric, satisfies the following probability law for s = 0, 1, . . . , 2 d ~ 1 — 1: 

d-l d-l 

Pr(Z = 1 + af aj ) = Fr ( Z = E a f ai) 

3=1 ' 3=1 

d-l d-l 

= Pr(Z = ^2 af a j \ X = 0) Pr(A = 0) + Pr(Z = ^ af a j \ X = 1) Pr(A = 1) 

3=1 ' 3=1 



Ps +P 2 d-1- 



s 



2 

The entropy of Z is therefore 



H(z) = 2 2 "£ l p - +i y- i -' iog 2 



s=0 

2 d ~ 1 -l 



Ps + P 2 d -1- 



= 1+ E (p S +^-i- s )log 2 (-— ; ) (36) 

S=Q \Ps+P2 d -l-s/ 

where the last transition follows from the equality Yll^o 1 Ps = 1- 

We now derive an upper bound on the entropy of the random vector Z. To this end, let 

d-i 

Z i = e i + Y,®i,3<* j , * = l,2,...,n. (37) 

3=1 

Denoting 3>j = ($j i, &i,2, ■ ■ ■ , &i,d-i), we have that = (@i, 02, ■ ■ ■ , @ n ) is a random vector over 
{0,1}", and * = (*i,* 2 ,...,* n ) is a random vector over {0, 1}^ i) n , From the decomposition 
of Zi in (37), it follows from (35) that 

Pv($ i = (a < f\...,af 1 )) = p s + p 2 4_ 1 _ s , i = l,2,...,n, s = 0, 1, . . . , 2 d " 1 - 1. (38) 

From (37) and (38), and the same chain of equalities leading to (16), it follows that 



d-l 



H(Z) = n Yl (^+^-i- s )log 2 (-— ^ ) 

s=0 \Ps+P2d-l- s J 

+H(e 1 ,e 2 ,...,e n \ * 1 ,* 2 ,...,* n ). (39) 

As in the proof of Proposition 3.1, we define the syndrome as S = @H T where H is a parity-check 
matrix of the code C. As before, the transmitted codeword x € C only affects the ©-components of 
the vector Z in (37). In parallel to (17), we obtain 

#(0i, 2 ,..., © n | *1, *2, • • • , *„) <nR + H(S | *!,* 2 ,..., * n ). (40) 

Considering a parity-check equation which involves k variables, let {ii, i 2 , . . . , i^} be the set of 
indices of the variables involved in this parity-check equation. The component of the syndrome 
S which refers to this parity-check equation is zero if and only if the components of the sub- 
vector (0^, @j 2 , . . . , @j fc ) differ from the components of the sub- vector (X^, Aj 2 , . . . , Xi k ) in an 
even number of indices. It is clear from (35) that for an index i, where <&j = (a^f \ af , . . . , af^ 
for some s = 0, . . . , 2 d ~ 1 — 1, the random variables X{ and ©j are different in probability - ^p' 1-8 ■ 



Lemma 3.3. Given that $>i 2 , ... , &i k ) has k s elements of the type (a^a^, . . . , a^-i) (where 
s = 0, . . . , 2 d ~ 1 — 1), the probability that the components of (0^ , Oj 2 , . . . , ©« fc ) and (A^ , Xj 2 , . . . , Xi k ) 
differ in an even number of indices is equal to 



2<«-i_i 



i+ n i- 



s=0 



Ps+P 2 d-l- 



Proof. The lemma is proved in Appendix B.l. 



□ 



Based on Lemma 3.3 and the discussion above, it follows that given that the vector (&i L , &i 2 , ■ ■ ■ , &i k ) 
has k s elements of the type (a^\ a 2 s \ ■ . ■ , a^-i) then 



H(Si | & h ,& i2 ,...,& ik ) = h 



1 



2 o-i_i 



i- n 1 



s=0 



1-s 



Ps+P2d-l-s 



For a component Si (1 < « < n(l — i?)) of the syndrome S which refers to a parity-check equation 
involving A; variables 

H(Si | * 1 ,* 2 ,...,* n ) 
= | * il ,* i2 ,...,* ifc ) 

^{o.i}^- 1 )* 

=, £ U,...l-.- 1 ) 2 nV, + ^-,-,)*- 

«0>— >* 2 d-l-l U 



E s k 3 —k 



■ho 



2«-i_i 

i- n 



s=0 



1-s 



p s +p 2 d-l- 



The number of parity-check equations involving k variables is ra(l — R)dk, hence 

£T(S | *i,* 2 ,...,* n ) 

n(l-R) 

< £ fl"0Si|*i,* 2 ,...,*n) 



i=l 



£,_ i Go.....l-.J 2 n 1( — 



-i- s ; 



E s k a —k 



h 2 



2 £ t -l_ 1 



i- n i- 



s=0 



2p 2 d-i- s 

Ps + P 2 d-1- 



(41) 



By combining (39)-(41), an upper bound on the entropy of the random vector Z follows: 



2 d ~ 1 -l 

H(Z)<nR + n V (p s + p 2d _ 1 _ s ) log 2 ( — — ^ ) 



+n(i - R ) y: U e L „ x ) n (p. + ^-i-j* 



• I x 



i- n (i 



s=0 



(42) 



The substitution of (36) and (42) in (12) finally provides the lower bound on the conditional entropy 
H(X | Y) in (34). □ 



Discussion: The calculation of the lower bound in the RHS of (34) becomes more complex 
as the value of d is increased. However, for optimally chosen quantization levels, we show that 
as the value of d is increased (thus, increasing the number of quantization levels), this bound is 
monotonically increasing. To this end, let d > 2 be an arbitrary integer, and let (l[ d \ ■ ■ ■ , 4d-i_i) 
and their symmetric values around zero denote the optimal choice for a 2 d -level quantization. Let 
Pq\p\ \ ■ ■ ■ jP^-i denote the transition probabilities, as defined in (33), which are associated with 
the optimal 2 d quantization levels. In order to show the above monotonicity property, we prove in 
Appendix B.2 that there exist sub-optimal 2 d+1 quantization levels l\, ■ ■ ■ , ^2 d -i (together with 
their symmetric values around zero) so that even with this sub-optimal 2 d+1 -level quantization, 
the bound in the RHS of (34) is already better than the one which is calculated from the optimal 
choice of a 2 d -level quantization. 

Theorem 3.2 ('^-Level Quantization" Lower Bound on the Asymptotic Parity-Check 
Density of Binary Linear Block Codes). Let {C rn } be a sequence of binary linear block codes 
achieving a fraction 1 — e of the capacity of an MBIOS channel with vanishing bit error probability. 
Then, the asymptotic density of their parity-check matrices satisfies 



lim inf A m > 



1-e 



(43) 



where 



K x = K 2 In 



1 



1-C 



21n(2) C 



K 2 = — 



1-C 



cm E 



(Pi - P 2 d-i-i)' 
Pi+P2 d -l-i 



(44) 



Here, d > 2 is an arbitrary integer and the probabilities {pi} are introduced in (33) in terms 
of h > ••• > l 2 d-i-i G The optimal vector of quantization levels (h, ■ ■ ■ ,l 2 d-i_i) is given 

implicitly by the set of 2 d ~ 1 — 1 equations 



+ e h v\-\ 



(Pi + P2 d -l-i) 2 (Pi-1 + P2 d -i) 2 

where such a solution always exists. 3 



l,...,2 d_1 - 1. 



(45) 



See the footnote to Theorem 3.1 in p. 10. 



Proof. For an arbitrary sequence of binary linear block codes {C m } which achieves a fraction 1 — e to 
capacity with vanishing bit error probability, we get from Lemma 3.1 that lirrim^oo — — ^ ■ Ym ~* = 
where X m and Y m designate the transmitted codeword in the code C m and the received sequence, 
respectively, and n m designates the block length of the code C m . From Proposition 3.2, we obtain 



HCX. m . Y 



m \ * ml 



> 1-C-(1-Rn 



k 



E E L fca i ) II fe+^-l- s ) fcs 

fc I V ''''' 2 ^ *=° 

Es ^s=fc 
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i- n i- 



s=0 



2p 2 d_ 



p s + P 2 rf-i- 



Similarly to the proof of Theorem 3.1, by letting m tend to infinity and using the upper bound on 
^2(0 from Lemma 3.2, we get 



i-c-(i-(i- £ )c)x;U e L \ dl ) n 



1 



ct— 1 1 

n 



21n2 \ fj^ PS+P2J-1- 



< 0. 



Since Y^k^k = 1 and Yls=o p s = 1, the sum in the LHS of (46) is equal to 
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2 In 2 \ ^ Ps+P2t-i-s 



(46) 



(47) 



where or = J2k designates the asymptotic average right degree, and the last transition follows 
from Jensen's inequality. Combining (46) and (47) gives 



1 -C - (1 - (l-e)C) 



1 - 



2 In 2 
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or" 



s=0 



+ P 2 <*-1- 



< 0. 



This yields the following lower bound on the asymptotic average right degree: 

a K >K[+K 2 In Q 



(48) 




K' 2 = - 

By combining (28) and (48) with the asymptotic rate R = (1 — e)C, we obtain a lower bound on 
the asymptotic parity-check density which is of the form 

K 1 + K 2 ]n(±) 
l-e 

where 

Kit = ^ ■ K[,2- 

This completes the proof of the lower bound in (43) and (44). The derivation of the set of opti- 
mization equations in (45) follows along the lines of the derivation of (22). In the general case of 
2 d quantization levels, it follows from (44) that we need to maximize 



E 



P 2 d-l- s ) 



„ Ps+P 2 d-l-s 

To this end, we set to zero all the partial derivatives w.r.t. l s where s = 1, . . . , 2 d ~ 1 — 1. Since from 
(33) only p s , p s _i, P2 d - S an d P2 d - s -i depend on l s , then 

d f (ps-i -P2*-s) 2 {Ps-P2*-s~lf ■ ( , 



[ Ps-l+P2d-s Ps+P2d-s-l 

We express now the probabilities p s , p s -\, P2 d -s an d P2 d -s-i as integrals of the conditional pdf 
a(-) of the LLR, and rely on the symmetry property where a(l) = e l a(—l) for l£l. In a similar 
manner to the derivation of (22), this gives the set of equations in (45). Their solution provides 
the quantization levels h, . . . , l 2 <i-i_i (where according to Proposition 3.2, the other 2 d ~ 1 — 1 levels 
are set to be symmetric w.r.t. zero). □ 



Based on the proof of Theorem 3.2, we derive an upper bound on the asymptotic rate of every 
sequence of binary linear codes for which reliable communication is achievable. The bound refers 
of soft-decision ML decoding, and it is therefore valid for any sub-optimal decoding algorithm. 

Corollary 3.2 ("2 d -Level Quantization" Upper Bound on the Asymptotic Achievable 
Rates of Sequences of Binary Linear Block Codes). Let {C m } be a sequence of binary linear 
block codes whose codewords are transmitted with equal probability over an MBIOS channel, and 
suppose that the block length of this sequences of codes tends to infinity as m — > oo. Let dk, m be 
the fraction of the parity-check nodes of degree k in an arbitrary representation of the code C m 
by a bipartite graph. Then a necessary condition for this sequence to achieve vanishing bit error 
probability as m — > oo is that the asymptotic rate R of this sequence satisfies 



R < 1 - max < (1 - C) 



i=0 
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k 



d, 
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■Eiki= 
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/uQ ; . . . , Iv^d — l . 



i- n (i- 



2P2 d -l-i 

Pi +p 2 d-l- 



n (Pi+P2d-i-d h 



2 d -l 

2 ^ 

i=2 d ~ 1 



2 d -l 



l-J>(l-2 E Pi)' 



i=2 d - 1 



(49) 



where d > 2 is arbitrary, the probabilities {pi} are introduced in (33), and dk and R are introduced 
in(l). 



Proof. The concept of the proof is the same as the proof of Corollary 3.1, except that the first term 
in the RHS of (49) relies on (34). □ 



4 Approach II: Bounds without Quantization of the LLR 

Similarly to the previous section, we derive bounds on the asymptotic achievable rate and the 
asymptotic parity-check density of an arbitrary sequence of binary, linear block codes transmitted 
over an MBIOS channel. As in Section 3, the derivation of these two bounds is based on a lower 
bound on the conditional entropy of a transmitted codeword given the received sequence at the 
output of an arbitrary MBIOS channel. 

Proposition 4.1. Let C be a binary linear code of length n and rate R transmitted over an 
MBIOS channel. Let x = (x±, x%, . . . , x n ) and y = (yi, 1/2, ■ ■ ■ , y n ) be the transmitted codeword and 
the received sequence, respectively. For an arbitrary representation of the code C by a parity-check 
matrix, let d^ designate the fraction of the parity-check equations of degree k. Then the conditional 
entropy of the transmitted codeword given the received sequence satisfies 



H(X\Y) 



n 



* 1 - ~ (1 - R) i 1 ~ 2hT(2i t ^h) £ * (T a( " (1 + e ~' ] tan " 2P G) dl ) 



(50) 

where a(-) denotes the conditional pdf ot the LLR given that the transmitted symbol is zero. 

Proof. We consider a binary linear block code C of length n and rate R whose transmission takes 
place over an MBIOS channel. Let us assume that x £ C is the transmitted codeword, and the 
'0' and '1' symbols of the codeword are mapped to +1 and —1, respectively. The input alphabet 
to the channel is {+1,-1}, and the LLR as a function of the observation y at the output of the 
MBIOS channel gets the form 

LLR (y) = In — — — , y £ R. 

\PY\x(y\X = -I) J 

For the continuation of the proof, we move from the mapping of the MBIOS channel X — ► Y 
to an equivalent representation of the channel X — > Y, so that the conditional entropies of the 
transmitted codeword given the received sequences at the outputs of both channels are equal, i.e., 
H(X\Y) = H(X\Y). The basic idea for showing the equivalence between the original channel and 
the one which will be introduced shortly is based on the following two facts: Firstly, the LLR forms 
a sufficient statistics of the channel, and secondly, denoting the output of an MBIOS channel by 
y then, from the symmetry of the channel, LLR(— y) = — LLR(y). This means that the absolute 
value of the LLR doesn't change when the channel output is flipped, but then the sign of the LLR 
alternates. 

For the characterization of the equivalent channel, let a(-) designate the conditional pdf of 
the LLR given that the transmitted symbol is (i.e., given that the channel input is 1). For 
i = 1, 2, . . . , n, we randomly generate an i.i.d. sequence {k}f =1 w.r.t. the conditional pdf a(-), and 



define 



w% = Oi — < 



+1 if k>0 

-1 if k<0 

k ±1 w.p. \ if = 



The output of the equivalent channel is defined to be the sequence y = (y\, . . . , y n ) where 

Vi = (4>i,uji), i = l,2,...,n 

and 4>i = OiXi. The output of this equivalent channel at time i is therefore the pair (</>j,u;j) where 
<fii G {+1,-1} and uji £ IR + . This defines the memoryless mapping 

X -» Y = ($,fi) 

where $ is a binary random variable which is affected by X, and fi is a non-negative random 
variable which represents the absolute value of the LLR and whose pdf is 

[ a{u)) + a(-u) = (1 + e^) a{uj) if u > 
= { a(0) if 07 = • (51) 

We note that the transition in case to > follows from the symmetry property of a(-), and the 
random variable Q is clearly statistically independent of X. Following the lines which lead to (12), 
we obtain 

H(X\Y) >nR- H(Y) + nH(Y) - nC. (52) 

In order to get a lower bound on iJ(X|Y), we will calculate exactly the entropy of Y and obtain 
an upper bound on the entropy of Y. The calculation of the first entropy is direct 

H(Y) = H($,n) 

= H(Q) + H(§\n) 

= H(n) + E u ,[H(<S>\n = oj)] 

= H(Q) + 1 (53) 

where the last transition is due to the fact that given the absolute value of the LLR, its sign is 
equally likely to be positive or negative. We note that H(Q) is not expressed explicitly as it will 
cancel out later. 

We will now derive an upper bound on H(Y). 

H(Y) = #(($!,..., $ n ), («!,...,$}„)) 

= H(n 1 ,...,n n ) + H(($ 1 ,...,<!> n ), i (fii,...,^)) 

= nH(n) + #(($!, ...,$„), I (fii, . . . , fl n )) . (54) 

Let us introduce the assignment / : {+1, —1} — > {0, 1} where +1 and —1 are mapped back to 
and 1, respectively, and define $j = /($j). Since $j = ®iXi, then we obtain 

$i = Gi + Xj, i = l, ...,n 



where the last addition is modulo-2. Define the syndrome vector 

S =($!,..., ® n )H T 



where H is an arbitrary parity-check matrix of the binary linear block code C, and let L be the 
index of the vector ($1, . . . , $> n ) in the coset which corresponds to S. Since each coset has exactly 
2 nR elements which are equally likely then H (L) = nR, and we get 

#(($!,...,$„), | (ftx,...,^)) = H(S,L\ («!,...,«„)) 

< #(L) + #(S | (fii, . . .,$!„)) 
= ni? + il(S | («!,..., 

n(l-fl) 

< ni?+ | (Q 1 ,...,n n )) (55) 

Since . . . , x n ) is the original transmitted codeword in C (i.e., before the conversion of the 
symbols of the transmitted codeword to ±1), then 

S = (5>i,..., $ n )H T 

= {e 1 ,...,e n )H T + (x 1 ,...,x n )H T 
= (e 1 ,...,e n )H T . 

Let us look at the j-th parity-check equation which involves k variables. Let us assume that the set 
of indices of the active variables in this parity-check equation is {ii, . . . , i k }. Then, the component 
Sj of the syndrome is equal to 1 if and only if there is an odd number of ones in the random vector 

...,e ifc ). 

Lemma 4.1. If the j-th component of the syndrome S involves k active variables in its parity-check 
equation whose indices are {h,i2, ■ ■ ■ , ik}, then 

k 



Pr(5j = 1 | (p, h ,. . . ,n ik ) = (ai, . . .,a k )) 



1 



n, 2e~ a 
1 
V 1 + e~ am 

m=l x 



(56) 



Proof. The lemma is proved in Appendix C.l. 



□ 



We therefore obtain from Lemma 4.1 that 



H(s j \(n il ,...,n ik ) = (ai,..., a fe )) = h 2 



1 



m=l x 



and by taking the statistical expectation over the k random variables , . . . , fij fc , we get 
#(^|(%,...,^J) 

m=l 



1 00 -i / poo \ i 



TJ fa(ai) daida 2 . ■ ■ da k (57) 



2 In 

where the equality in the last transition is proved in Appendix C.3. Hence, if d k designates the 
number of parity-check equations of degree k, then 

n(l-R) 

£ i^K^!,...,^)) 



n(l - R) 



1 



2E2 ?*5^nj(y a»w tah2! '(f) *• 



(58) 



By combining (51), (54), (55) and (58), we get the following upper bound on H(Y): 
H(Y) < nH(Q) + nR 

+n(l - R) 



1 1 / /*oo 



.(59) 



Finally, the equality in (53) and the upper bound on H(Y) given in (59) are substituted in the 
RHS of (52). This provides the lower bound on the conditional entropy iT(X|Y) given in (50), and 
completes the proof of this proposition. □ 

Remark 4.1. For the particular case of a BEC with erasure probability p, C = 1 — p, and the 
conditional pdf of the LLR is independent of the transmitted symbol. It is equal to 

a(Z)=pA (Z) + (l-p)A oo (0 

where A a (-) designates the Dirac delta function at the point a. We obtain from (50) 

#(X|Y) 



n 



>P-(l-R) 



l-J2d k (l-p) h 



(60) 



This lower bound on the conditional entropy for the BEC coincides with the result proved in [13, 
Eqs. (33) and (34)]. The result there was obtained by the derivation of an upper bound on the 
rank of He which is a sub-matrix of H whose columns correspond to the variables erased by the 
BEC. 



Discussion: Since the proof of Proposition 4.1 relies on the analysis of an equivalent channel, 
rather than a degraded (quantized) channel, it is suggested that the lower bound in the RHS of 
(50) should be tighter than the one in the RHS of (34). In order to prove this property, it is enough 
to show that for any integer d > 2 and any choice of quantization levels l±, . . . , l 2 d-i_i, we have 



1 - 



1 



21n(2) 



I e* or am g 




k K fcovifejd-l-l 
Ei ki=k 



k 

, k 2 d-l_l 



i=0 



2 d-l_ 

i- n 

i=0 



2p 2 d_ 



l-i 



Pi+P 2 d- 1 . 



(61) 



where po, ■ ■ ■ ,P2 d -i are the transition probabilities associated with h, . . . , l 2 d-i_i, as defined in (33). 
This inequality is proved in Appendix D.l, using the power series expansion of the binary entropy 
function, h 2 (-), derived in Appendix C.2. 

Theorem 4.1 ("Un-Quantized" Lower Bound on the Asymptotic Parity-Check Density 
of Binary Linear Block Codes). Let {C m } be a sequence of binary linear codes achieving a 
fraction 1 — e of the capacity C of an MBIOS channel with vanishing bit error probability. Then, 
the asymptotic density of their parity check matrices satisfies 



liminf A m > sup 

m - > °° x€(0,A] 



K 1 {x) + K 2 {x) In \ 



(62) 



where 



1-C 

c 



In 



-c) 



i^ 2 (x) 



1-C 1 



and 



A 



(1-e 



-l\2 



+ e 



di, e = 



21n(2) 



for a BEC 
otherwise 



(63) 



(64) 



Proof. From the lower bound on v J — - in Eq. (50) and Lemma 3.1 (see p. 10), we obtain that 
if {C m } is a sequence of binary linear block codes which achieves a fraction 1 — e of the channel 
capacity with vanishing bit error probability, then 



l-C-(l-(l-e)C) 



£4£ 



21n ( 2 ) k ^PPP- 1 ) Wo 



a(/)(l + e-')tanh 2 P 



G) 



< 0. 



(65) 



Since fcdfc = or is the average right degree, then from the convexity of the exponential function, 
we obtain by invoking Jensen's inequality that 



l-C-(l-(l-e)C) 



1 00 1 / /-oo 

ilS(2)E^ri) (J 



I f , -') laulr'' f ^ all" 



< 0. (66) 



We will now derive two different lower bounds on the infinite sum in the RHS of (66), and 
compare them later. For the derivation of the lower bound in the first approach, let us define the 
positive sequence 



a. 



, P=l,2,. 



(67) 



p ln(2) 2p(2p - 1) 

From (C.l) in the appendix, the substitution of x = in both sides of the equality gives that 



Yl^Li a p = 1) so the sequence {a p } forms a probability distribution. We therefore obtain that 

1 00 1 / poo 



00 / /-OO / 7 \ \ OR 

J2 a P yj a(0(l + e^)tanh 2p ^-J d/J 



(a) 
> 



(b) 



(c) 



/•OO 00 / 7 \ 

y a(0(l + e- / )£«ptanh 2p (^-J dZ 
l a(l)(l + e- 1 ) 



or 



1 - ho - 



1 — tanh ( - 



OR 



d/ 



or 



dl 



(£) c a R 



(68) 



where inequality (a) follows from Jensen's inequality, equality (b) follows from (67) and (C.l), 
equality (c) follows from the identity tanh (a;) = , and equality (d) follows from the relation 



between the capacity of an MBIOS channel and the pdf of the absolute value of the LLR (see [12, 
Lemma 3.13]). 

For an alternative derivation of the lower bound of the infinite series, we can truncate the 
infinite sum in the RHS of (66) and take into account only the first term in this series. This gives 

\ or 

21n(2) 



11 / f°° f i \ \ ' 

^)(r o(0(i+e "' )tmh2 (c 




1R 

dl ' 



1R 



21n(2) 

where the last transition follows from (64). 



(69) 



In order to compare the tightness of the two lower bounds in (68) and (69), we first compare 
the bases of their exponents (i.e., A and C). To this end, it is easy to verify that 



-i\ 2 
e 1 x 



1 + e 



-l 



with an equality if and only if / = or I — > oo. Hence, from (68) and (69), this gives A > C with 
equality if and only if the MBIOS channel is a BEC. Therefore, up to the multiplicative constant 
2^2j, the second lower bound is tighter than the first one. However, we note that for the BEC, 
the first bound is tighter. It gives an improvement by a factor of 21n(2) 1.386. 

We will therefore continue the analysis based on the second bound in (69), and then give the 
potential improvement which follows from the first bound in (68) for a BEC. From (66) and (69), 
we obtain that 



/ 4QR \ 



Hence, one can replace A (where < A < 1) in the last inequality by an arbitrary x € (0, A], and 
obtain that the asymptotic average right degree, or, satisfies the lower bound 

" R - Ml) • 

By dropping the 1 inside the logarithm in the numerator, we obtain that for x G (0, A] 

a R >K' 1 (x)+K^(x)]n(^j (70) 



1 l-C 1 



where K[(x) = ^(i) and K' 2 {x) = Finally, since the parity-check density and 

average right degree are related by the equality A = (^jp) «r, then we obtain the following lower 
bound on the asymptotic parity-check density: 

hminfA m > 1 ~ {1 ~ S J 1 C (K[(x)+K' 2 (x)\n(-)) 

m-*oo (1 — E)C \ \£ / / 

KAx) +K 2 (x)ln(±) , n 

> V 7 - V 7 VxE(0,A] (71) 



where K\^{x) = ^-jf- K[ 2 (x). For the BEC, this lower bound can be improved by using the first 
bound in (68). In this case, A = C = 1 — p where p designates the erasure probability of the BEC, 
so the additive coefficient K\ in the RHS of (62) is improved to 

P 

K 1 (x) = -Z- K *' , x€(0,l-p. 
1-P In(-) 

This concludes the proof of this theorem. □ 

Remark 4.2. For a BEC with erasure probability p, the maximization of the RHS of (62) yields 
that the maximal value is achieved for x = 1 — p when e < (otherwise, if e > the lower 
bound is useless as it becomes non-positive). Hence, the lower bound on the asymptotic parity- 
check density stated in Theorem 4.1 coincides with the bound for the BEC in [13, Eq. (3)]. This 
lower bound was demonstrated in [13, Theorem 2.3] to be tight. This is proved by showing that 
the sequence of right-regular LDPC ensembles of Shokrollahi [14] is optimal in the sense that it 
achieves (up to a small additive coefficient) the lower bound on the asymptotic parity-check density 
for the BEC. 

For a general MBIOS channel (other than the BEC), we show in the proof above that the 
preferable logarithmic growth rate of the lower bound on the parity-check density is achieved by 
using the bound which follows from (69) (even in the particular case where x = A). However, we 
note that the lower bound on the parity-check density which follows from (68) is universal w.r.t. 
all MBIOS channels with the same capacity. 

Remark 4.3. The lower bound on the parity-check density in Theorem 4.1 is uniformly tighter 
than the one in [13, Theorem 2.1] (except for the BSC and BEC where they coincide). For a proof 
of this claim, the reader is referred to Appendix D.2. 

Based on the proof of Theorem 4.1, we prove and discuss an upper bound on the asymptotic rate 
of every sequence of binary linear codes for which reliable communication is achievable. The bound 
refers of optimal ML decoding, and is therefore valid for any sub-optimal decoding algorithm. Hence 
the following result also provides an upper bound on the achievable rate of ensembles of LDPC 
codes under iterative decoding, where the transmission takes places over an MBIOS channel. 

Corollary 4.1 ("Un-Quantized" Upper Bound on the Asymptotic Achievable Rates of 
Sequences of Binary Linear Block Codes). Let {C m } be a sequence of binary linear block 
codes whose codewords are transmitted with equal probability over an MBIOS channel, and assume 
that the block lengths of these codes tend to infinity asra-> oo. Let d}~ m be the fraction of the 
parity-check nodes of degree k for arbitrary representations of the codes C m by bipartite graphs. 
Then a necessary condition on the achievable rate (R) for obtaining vanishing bit error probability 
as m — > oo is 

*<l : ^ : ^ 7- rrv (72) 



21n(2) p , 
where and R are introduced in (1). 



£ i^by £ * Of 0(0 {l+e '' ] tanh * G) ' 



Proof. This upper bound on the achievable rate follows immediately from Lemma 3.1 (see p. 10) 
and the lower bound on the conditional entropy in Proposition 4.1. The upper bound on R follows 
since the bit error probability of the sequence of codes {C m } vanishes as we let m tend to infinity. □ 



Remark 4.4. We note that the upper bound on the achievable rate in the RHS of (72) doesn't 
involve maximization, in contrast to the bound in the RHS of (49). The second term of the 
maximization in the latter bound follows from considerations related to the BEC where such an 
expression is not required in the RHS of (72). The reader is referred to Appendix D.3 for a proof 
of this claim. 

Corollary 4.2 (Lower Bounds on the Bit Error Probability of LDPC Codes). Let C be 

a binary linear block code of rate R whose transmission takes place over an MBIOS channel with 
capacity C. For an arbitrary parity-check matrix H of the code C, let dk designate the fraction of 
parity-check equations that involve k variables. Then, under ML decoding (or any other decoding 
algorithm), the bit error probability (Pt,) OI the code satisfies 

^ 2 1 - 1 + £ [ E * Of 0(0(1 + e " ] tanh2P (£> '") '} ' (73> 

Proof. This follows directly by combining (23) and (50). □ 

We now introduce the definition of normalized parity-check density from [13], and derive an 
improved lower bound on the bit error probability (as compared to [13, Theorem 2.5]) in terms of 
this quantity. 

Definition 4.1 (Normalized parity-check density [13]). Let C be a binary linear code of rate 
R, which is represented by a parity-check matrix H whose density is A. The normalized density of 
H, call it t = t(H), is defined to be t = 

In the following, we clarify the motivation for the definition of a normalized parity-check density. 
Let us assume that C is a binary linear block code of length n and rate R, and suppose that it can 
be represented by a bipartite graph which is cycle-free. From [13, Lemma 2.1], since this bipartite 
graph contains (2 — R)n — 1 edges, connecting n variable nodes with (1 — R)n parity-check nodes 
without any cycles, then the parity-check density of such a cycle- free code is A = — Hence, 
in the limit where we let n tend to infinity, the normalized parity-check density of a cycle-free code 
tends to 1. For codes which are represented by bipartite graphs with cycles, the normalized parity- 
check density is above 1. As shown in [13, Corollary 2.5], the number of fundamental cycles in 
a bipartite graph which represents an arbitrary linear block C grows linearly with the normalized 
parity-check density. The normalized parity-check density therefore provides a measure for the 
number of cycles in bipartite graphs representing linear block codes. It is well known that cycle- 
free codes are not good in terms of performance, even under optimal ML decoding [15]; hence, good 
error-correcting codes (e.g., LDPC codes) should be represented by bipartite graphs with cycles. 
Following the lead of [13], providing a lower bound on the asymptotic normalized parity-check 
density in terms of their rate and gap to capacity gives a quantitative measure for the number of 
fundamental cycles of bipartite graphs representing good error correcting codes. In the following, 
we provide such an improved bound as compared to the bound given in [13, Theorem 2.5]. In the 
continuation (see Section 5.2), the resulting improvement is exemplified. 

First, we note that from Definition 4.1, it follows that the relation between the normalized 
parity-check density and the average right degree is 

1-R\ 
2^r) aR 

so the normalized parity-check density grows linearly with the average right degree (which is directly 
linked to the decoding complexity per iteration of LDPC codes under message-passing decoding) 
where the scaling factor depends on the code rate R. 



Since J^k^dk = or, then by applying Jensen's inequality to the RHS of (73), we get the 
following lower bound on the bit error probability: 



h 2 (Pb) > 1 



C 1 

+ 



R 



E 



R 21n(2)i? ^ I p(2p- 1) 



P =i 



a(/)(l + e-')tanh 2p 



dl 



(2-R)t 
1-R 



(74) 



This lower bound on the bit error probability is tighter than the bound given in [13, Eq. (23)] because 
of two reasons: Firstly, by combining inequality (69) with the inequality proved in Appendix D.2, 
we obtain that 



1 



21n(2) 



E 

p=i 



p(2p 



dl 



(2-Ryt 

l-R 



> 



2(2-fl)t 

2 In 2 



Secondly, the further improvement in the tightness of the new bound is obtained by dividing the 
RHS of (74) by R (where R < 1), as compared to the RHS of [13, Eq. (23)]. 

The bounds in (73) and (74) become trivial when the RHS of these inequalities are non-positive. 
Let the (multiplicative) gap to capacity be defined as e = 1 — ^. Analysis shows that the bounds 
in (73) and (74) are useful unless e > £q (see Appendices D.4 and D.5). For the bound in the RHS 
of (73), £q gets the form 



(l-C)B 
C(l-B) 



D 



5 M 



(75) 



and for the bound in the RHS of (74), eq is the unique solution of the equation 
l-(l-e )C 



-E C + 



21n(2) 



E 

P =i 



p(2p - 1) 



a(0(l + O 



tanh 2 f (I 



dl 



(2-(l-e )C)t ^ 
l-(l-e )C 



> = 0. (76) 



For a proof of (75) and (76), the reader is referred to Appendices D.4 and D.5, respectively. 
Similarly to [13, Eq. (25)], we note that Eq in (76) forms a lower bound on the gap to capacity for 
an arbitrary sequence of binary linear block codes achieving vanishing bit error probability over 
an MBIOS channel; the bound is expressed in terms of their asymptotic rate R and normalized 
parity-check density t. It follows from the transition from (73) to (74) that the lower bound on 
the gap to capacity in (76) is looser as compared to the one given in (75). However, the bound in 
(76) solely depends on the normalized parity-check density, while the bound in (75) requires full 
knowledge of the degree distribution for the parity-check nodes. 



5 Numerical Results 



In this section we present numerical results for the information-theoretic bounds on the limitations 
of binary linear block codes transmitted over MBIOS channels. These results refer to Theorems 3.1, 
3.2 and 4.1 and Corollaries 3.1, 3.2, 4.1 and 4.2. As expected, they significantly improve the 
numerical results presented in [1, Section 4] and [13, Section 4]. This improvement is attributed to 
the fact that, in contrast to [1, 13], in the derivation of the bounds in this paper, we do not perform 
a two-level quantization of the LLR which in essence converts the arbitrary MBIOS channel (whose 
output may be continuous) to a BSC. Throughout this section, we assume transmission of the codes 
over the binary-input AWGN channel. 

5.1 Thresholds of LDPC Ensembles under ML Decoding 

The following results (see Tables 1-3) provide bounds on the thresholds of LDPC ensembles under 
ML decoding. They also give an indication on the inherent loss in performance due to the sub- 
optimality of iterative message-passing decoding. 



LDPC 
Ensemble 


Capacity 
Limit 


2-Levels 
Bound [1] 


4-Levels 
Bound 


8-Levels 
Bound 


Un-Quantized 
Lower Bound 


Upper 
Bound [5] 


DE 

Threshold 


(3,6) 


+0.187 dB 


+0.249 dB 


+0.332 dB 


+0.361 dB 


+0.371 dB 


+0.673 dB 


+ 1.110 dB 


(4,6) 


-0.495 dB 


-0.488 dB 


-0.472 dB 


-0.463 dB 


-0.463 dB 


-0.423 dB 


+1.674 dB 


(3,4) 


-0.794 dB 


-0.761 dB 


-0.713 dB 


-0.694 dB 


-0.687 dB 


-0.510 dB 


+1.003 dB 



Table 1: Comparison of thresholds for Gallager's ensembles of regular LDPC codes transmitted 
over the binary-input AWGN channel. The 2-level lower bound on the threshold of refers to 
ML decoding, and is based on [1, Theorem 1] (see also [13, Table II]). The 4- level, 8-level and 
un-quantized lower bounds apply to ML decoding, and are based on Corollaries 3.1, 3.2 and 4.1, 
respectively. The upper bound on the threshold of holds under 'typical pairs' decoding [5] (and 
hence, also under ML decoding), and the DE thresholds are based on density evolution for iterative 
message-passing decoding [10]. 

The upper bounds on the achievable rates derived in [1] and Corollaries 3.1, 3.2 and 4.1 provide 
lower bounds on the jP- thresholds under ML decoding. For Gallager's regular LDPC ensembles, 
the gap between the thresholds under ML decoding and the exact thresholds under the sum-product 
decoding algorithm (which are calculated using density-evolution analysis) are rather large. For 
this reason, we also compare the lower bounds on the thresholds under ML decoding with upper 

bounds on the thresholds which rely on "typical pairs decoding" [5]; an upper bound on the jf 2 - 
thresholds under an arbitrary sub-optimal decoding algorithm (e.g., "typical pairs decoding") also 
forms an upper bound on these thresholds under optimal ML decoding. It is shown in Table 1 that 
the gap between the thresholds under iterative decoding and the bounds for ML decoding (see the 
columns referring to the DE threshold and the upper bound based on "typical pairs decoding") 
is rather large. This is attributed to the sub-optimality of belief propagation decoding for regular 
LDPC ensembles. On the other hand, it is also demonstrated in Table 1 that the gap between 
the upper and lower bounds on the thresholds under ML decoding is much smaller. For example, 
according to the numerical results in Table 1, the inherent loss in the asymptotic performance due 
to the sub-optimality of belief propagation for Gallager's ensemble of (4, 6) regular LDPC codes 



(whose design rate is | bits per channel use) ranges between 2.097 and 2.137 dB. 

For irregular LDPC ensembles, the calculation of similar upper bounds based on "typical pairs" 
decoding [5] is based on the calculation of the asymptotic growth rate of the distance spectra of such 
ensembles. For a given pair of degree distributions (A, p), the calculation of the asymptotic growth 
rates of the distance spectrum for the (n, A, p) LDPC ensemble (where we let n tend to infinity) 
is tractable (see [2, 17]). However, we avoid calculating these upper bounds in Tables 2 and 3 due 
to the fact that the gap between the DE thresholds under belief propagation and the improved 
lower bounds on the j^- thresholds derived in this paper is already rather small (see Tables 2 and 
3). The rather small gap between the DE thresholds and the un-quantized lower bounds on the 
thresholds under ML decoding also indicate that for the degree distributions which are provided by 
the LDPC optimizer [16], the asymptotic degradation in performance due to the sub-optimality of 
belief propagation is marginal (it is observed from Tables 2 and 3 that for several LDPC ensembles, 
this degradation in the asymptotic performance is at most in the order of hundredthes of a decibel). 



A(x) 


p(x) 


2-Levels 
Bound [1] 


4-Levels 
Bound 


8-Levels 
Bound 


Un-Quantized 
Lower Bound 


DE 

Threshold 


0.38354x + 0.04237x 2 + 
0.57409a; 3 


0.24123x 4 + 
0.75877x 5 


0.269 dB 


0.370 dB 


0.404 dB 


0.417 dB 


0.809 dB 


0.23802x + 0.20997x 2 + 
0.03492x 3 + 0.12015x 4 + 
0.01587x 6 +0.00480x 13 + 
0.37627a; 14 


0.98013x 7 + 
0.01987x 8 


0.201 dB 


0.226 dB 


0.236 dB 


0.239 dB 


0.335 dB 


0.21991a; + 0.23328a; 2 + 
0.02058a; 3 + 0.08543a; 5 + 
0.06540a; 6 + 0.04767a; 7 + 
0.01912x 8 +0.08064x 18 + 
0.22798a; 19 


0.64854x 7 + 
0.34747x 8 + 
0.00399x 9 


0.198 dB 


0.221 dB 


0.229 dB 


0.232 dB 


0.310 dB 


0.19606a; + 0.24039x 2 + 
0.00228a; 5 + 0.05516a; 6 + 
0.16602a; 7 + 0.04088a; 8 + 
0.01064x 9 + 0.00221x 27 + 
0.28636a; 29 


0.00749x 7 + 
0.99101x 8 + 
0.00150x 9 


0.194 dB 


0.208 dB 


0.214 dB 


0.216 dB 


0.274 dB 



Table 2: Comparison of thresholds for rate one-half ensembles of irregular LDPC codes transmitted 
over the binary-input AWGN channel. The Shannon capacity limit corresponds to jf- = 0.187 dB. 
The 2-level, 4-level, 8-level and un-quantized lower bounds on the threshold refer to ML decoding, 
and are based on [1, Theorem 2], Corollaries 3.1, 3.2 and 4.1, respectively. The degree distributions 
of the ensembles and their DE thresholds are based on density evolution under iterative message- 
passing decoding [10], and are taken from [11, Tables 1 and 2]. 

The plots in Figure 2 compare different lower bounds on the ^--threshold under ML decoding 
of right-regular LDPC ensembles. The plots refer to a right degree of 6 (left plot) or 10 (right plot). 
The following lower bounds are depicted in these plots: the Shannon capacity limit, the 2-level 
quantization lower bound in [1, Theorem 1], the 4 and 8-level quantization bounds of the LLR in 
Section 3, and finally, the bound in Section 4 where no quantization of the LLR is performed. It 
can be observed from the two plots in Figure 2 that the range of code rates where there exists a 
visible improvement with the new lower bounds depends on the degree of the parity-check nodes. 
In principle, the larger the value of the right-degree is, then the improvement obtained by these 
bounds is more pronounced starting from a higher rate code rate (e.g., for a right degree of 6 or 



A(x) 


p{x) 


2-Levels 
Bound [1] 


4-Levels 
Bound 


8-Levels 
Bound 


Un-Quantized 
Lower Bound 


DE 

Threshold 


0.302468x + 0.319447x 2 + 
0.378085x 4 


x 11 


1.698 dB 


1.786 dB 


1.815 dB 


1.825 dB 


2.049 dB 


0.244067x + 0.292375x 2 + 
0.463558x 6 


x 13 


1.664 dB 


1.718 dB 


1.736 dB 


1.742 dB 


1.874 dB 


0.205439x + 0.255432x 2 + 
0.0751187x 4 +0. 1013440x 5 + 
0.3626670X 11 


x 15 


1.647 dB 


1.680 dB 


1.691 dB 


1.695 dB 


1.763 dB 



Table 3: Comparison of thresholds for rate-| ensembles of irregular LDPC codes transmitted over 
the binary-input AWGN channel. The Shannon capacity limit corresponds to j^- = 1.626 dB. The 
2-level, 4-level, 8-level and un-quantized lower bounds on the threshold refer to ML decoding, and 
are based on [1, Theorem 2], Corollaries 3.1, 3.2 and 4.1, respectively. The degree distributions 
of the ensembles and their DE thresholds are based on density evolution under iterative message- 
passing decoding [10], and are taken from [16]. 



10, the improvement obtained by the new bounds is observed for code rates starting from 0.35 and 
0.55 bits per channel use, respectively). 



Right-Regular LDPC a R = 6 Right-Regular LDPC a R = 10 
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Figure 2: Comparison between different lower bounds on the threshold under ML decoding for 
right-regular LDPC ensembles with or = 6 (left plot) and or = 10 (right plot). The transmission 
takes place over the binary-input AWGN channel. 



5.2 Lower Bounds on the Bit Error Probability of LDPC Codes 

By combining the lower bound in Proposition 4.1 and Lemma 3.1, we obtain in Corollary 4.2 an 
improved lower bound on the bit error probability of binary linear block codes, as compared to the 
one given in [13, Theorem 2.5]. The plot of Fig. 3 presents a comparison of these lower bounds for 
binary linear block codes where the bounds rely on (74) and [13, Theorem 2.5]. They are plotted as 
a function of the normalized density of an arbitrary parity-check matrix. In our setting, the capacity 
of the channel is \ bit per channel use, and the bounds are depicted for binary linear block codes 
whose rate is a fraction 1 — e of the channel capacity. To demonstrate the advantage of the lower 



bound on the bit error probability in (74) over the lower bound derived in [13, Theorem 2.5], let 
us assume that one wants to design a binary LDPC code which achieves a bit-error probability of 
10~ 6 at a rate which is 99% of the channel capacity. The curve of the lower bound from [13] for 
e = 0.01 implies that the normalized density of an arbitrary parity-check matrix which represents 
the code (see Definition 4.1 in p. 27) should be at least 4.33, while the curve depicting the bound 
from (74) strengthens this requirement to a normalized density (of each parity-check matrix) of at 
least 5.68. Translating this into terms of parity-check density (which is also the complexity per 
iteration for message-passing decoding) yields minimal parity-check densities of 13.16 and 17.27, 
respectively (the minimal parity-check density is given by A m j n = — ^* mm ). It is reflected from 
Fig. 3 that as the gap to capacity e tends to zero, the lower bound on the normalized density of an 
arbitrary parity-check matrix (t), which represents a code which achieves low error probability for 
a rate of R = (1 — e)C grows significantly. 



Lower Bound on P in Corollary 4.2 (e = 0) 

Lower Bound on P b [13, Theorem 2.5] (e = 0) 

Lower Bound on P b in Corollary 4.2 (e = 0.01 ) 



Lower Bound on P [13, Theorem 2.5] (e = 0.01) 




1 2 3 4 5 6 7 8 

Normalized Parity-Check Density (I) 



Figure 3: Lower bounds on the bit error probability for any binary linear block code transmitted 
over a binary-input AWGN channel whose capacity is \ bits per channel use. The bounds are 
depicted in terms of the normalized density of an arbitrary parity-check matrix which represents 
the code, and the curves correspond to code rates which are a fraction 1 — e of the channel capacity 
(for different values of e). The bounds depicted in dashed lines are based on [13, Theorem 2.5], 
and the bounds in solid lines are given in Corollary 4.2. 



5.3 Lower Bounds on the Asymptotic Parity-Check Density 

The lower bound on the parity-check density derived in Theorem 4.1 enables to assess the tradeoff 
between asymptotic performance and asymptotic decoding complexity (per iteration) of an iterative 
message-passing decoder. This bound tightens the lower bound on the asymptotic parity-check 
density derived in [13, Theorem 2.1]. Fig. 4 compares these bounds for codes of rate \ (left plot) 
and | (right plot) where the bounds are plotted as a function of j^. It can be observed from 
Fig. 4 that as jj* increases, the advantage of the bound in Theorem 4.1 over the bound in [13, 

Theorem 2.1] diminishes. This follows from the fact that as the value of ^ is increased, the two- 
level quantization of the LLR used in [1] and [13, Theorem 2.1] better captures the true behavior 
of the MBIOS channel. It is also reflected in this figure that as e tends to zero (i.e., when the gap 
to capacity vanishes), the slope of the bounds becomes very sharp. This is due to the logarithmic 
behavior of the bounds. 
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Figure 4: Comparison between lower bounds on the asymptotic parity-check density of binary 
linear block codes where the transmission takes place over a binary-input AWGN channel. The 
dashed line refers to [13, Theorem 2.1], and the solid line refers to Theorem 4.1. The left and right 
plots refer to code rates of \ and f , respectively. The Shannon capacity limit for these code rates 
corresponds to of 0.187 dB and 1.626 dB, respectively. 

6 Summary and Outlook 

We derive improved lower bounds on the asymptotic density of parity-check matrices and upper 
bounds on the achievable rates of binary linear block codes transmitted over memoryless binary- 
input output-symmetric (MBIOS) channels. The improvements are w.r.t. the bounds given in 
[1, 13]. The information-theoretic bounds are valid for every sequence of binary linear block codes, 
in contrast to high probability results which follow from probabilistic analytical tools (e.g., density 
evolution (DE) analysis under iterative decoding). The bounds hold under optimal ML decoding, 
and hence, they hold in particular under any sub-optimal decoding algorithm. We apply the 
bounds to ensembles of low-density parity-check (LDPC) codes. The significance of the bounds is 
the following: firstly, by comparing the new upper bounds on the achievable rates with thresholds 
provided by DE analysis, we obtain rigorous bounds on the inherent loss in performance of various 
LDPC ensembles. This degradation in the asymptotic performance is due to the sub-optimality of 
iterative message-passing decoding (as compared to optimal ML decoding). Secondly, the parity- 
check density can be interpreted as the complexity per iteration under iterative message-passing 
decoding. Therefore, by tightening the reported lower bound on the asymptotic parity-check density 
[13, Theorem 2.1], the new bounds provide better insight on the tradeoff between the asymptotic 
performance and the asymptotic decoding complexity of iteratively decoded LDPC codes. Thirdly, 
the new lower bound on the bit error probability of binary linear block codes tightens the reported 
lower bound in [13, Theorem 2.5]. 

The derivation of the bounds in Section 3 was motivated by the desire to generalize the results in 
[1, Theorems 1 and 2] and [13, Theorem 2.1]. The two-level quantization of the log-likelihood ratio 
(LLR) which in essence replaces the arbitrary MBIOS channel by a physically degraded binary 
symmetric channel (BSC), is modified in Section 3 to a quantized channel which better reflects 
the statistics of the original channel (though the quantized channel is still physically degraded 
w.r.t. the original channel). The number of quantization levels at the output of the new channel 
is an arbitrary integer power of 2. The calculation of the bounds in Section 3 is subject to an 
optimization of the quantization levels of the LLR, as to get the tightest bounds within their form. 



In Section 4, we rely on the conditional pdf oi the LLR at the output of the MBIOS channel, and 
operate on an equivalent channel without quantizing the LLR. This second approach finally leads 
to bounds which are uniformly tighter than the bounds we derive in Section 3. It appears to be 
even simpler to calculate the un-quantized bounds in Section 4, as their calculation do not involve 
the solution of any optimization equation (in contrast to the quantized bounds, whose calculation 
involves a numerical solution of optimization equations w.r.t. the quantization levels of the LLR). 
The comparison between the quantized and un-quantized bounds gives insight on the effect of the 
number of quantization levels of the LLR (even if they are chosen optimally) on the achievable rates, 
as compared to the ideal case where no quantization is done. The results of such a comparison are 
shown in Tables 1-3, and indicate that the improvement in the tightness of the bounds when more 
than 8 levels of quantization are used (in case the quantization levels are optimally determined) is 
marginal. We also note that practically, the possibility to calculate un-quantized bounds which are 
uniformly better than the quantized bounds was facilitated due to an efficient transformation of the 
multi-dimensional integral in Appendix C.3 into an infinite series of one-dimensional integrals whose 
convergence rate is very fast (10 terms are sufficient in practice, as justified in Appendix C.3). Had 
we used instead an upper bound on ^(O like the one in Lemma 3.2 (see p. 10), 4 then this would 
loosen the un-quantized bounds, and make them sometimes even worse than the 8-level quantized 
bounds. The ability to express the A;-dimensional integral in Appendix C.3 in a closed form was 
therefore crucial for practically obtaining an un-quantized bound which is uniformly tighter than 
the quantized bounds for an arbitrary number of quantization levels. As mentioned before, the 
exact calculation of the un-quantized bound also provides insight on the effect of the number of 
quantization levels on the tightness of the quantized bounds (see Tables 1-3 in Section 5). 

Our bounds on the thresholds of LDPC ensembles under optimal ML decoding depend only on 
the degree distribution of their parity-check nodes and their design rate. For a given parity-check 
degree distribution (p) and design rate (R), the bounds provide an indication on the inherent gap 
to capacity which is independent of the choice of A (as long as the pair of degree distributions (A, p) 
yield the design rate R). Therefore, our bounds are not expected to be tight for LDPC ensembles 
with a given pair of degree distributions (A, p). The numerical results shown in Tables 1-3 indicate, 
however, that these bounds are useful for assessing the inherent gap to capacity of various (regular 
and irregular) LDPC ensembles. The comparison of our bounds with the DE thresholds (based on 
density evolution) provides an assessment of the degradation in the asymptotic performance due to 
the sub-optimality of the iterative sum-product decoding algorithm. We note that the gap of LDPC 
ensembles to capacity is an inherent phenomenon, due to their finite average-right degree [13]. As 
a topic for further research, it is suggested to examine the possibility of tightening the bounds for 
specific ensembles by explicitly taking into account the exact characterization of A. We also suggest 
to study a possible generalization of the bounds to non-binary linear block codes. These generalized 
bounds can be applied to the analysis of the ML performance of non-binary LDPC ensembles whose 
transmission takes place over arbitrary discrete memoryless channels with possibly different types 
of quantization [3]. 

The lower bound on the asymptotic parity-check density in [13, Theorem 2.1] and its improve- 
ments in Sections 3 and 4 grow like the log of the inverse of the gap (in rate) to capacity. The result 
in [13, Theorem 2.3] shows that a logarithmic growth rate of the parity-check density is achiev- 
able for Gallager's regular LDPC ensemble under ML decoding when transmission takes place over 
an arbitrary MBIOS channel. These results show that for any iterative decoder which is based 
on the representation of the codes by Tanner graphs, there exists a tradeoff between asymptotic 
performance and complexity which cannot be surpassed. Recently, it was shown in [9] that better 
tradeoffs can be achieved by allowing more complicated graphical models; for the particular case of 

4 This was actually the way we used to calculate the un-quantized bounds at the beginning, due to an initial 
difficulty in calculating the k dimensional integral in Appendix C.3. 



the binary erasure channel (BEC), the encoding and the decoding complexity of properly designed 
codes on graphs remain bounded as the gap to capacity vanishes. To this end, Pfister, Sason and 
Urbanke consider in [9, Theorems 1 and 2] ensembles of irregular repeat- accumulate codes which 
involve punctured bits, and allow in this way a sufficient number of state nodes in the Tanner graph 
representing the codes. This surprising result is considered in [9, Theorem 4], by a derivation of 
an information-theoretic lower bound on the decoding complexity of randomly punctured codes 
on graphs whose transmission takes place over MBIOS channels. The approach for the derivation 
of the bounds in [9, Theorem 4] rely on the analysis in [13, Theorem 2.1]. As a topic for further 
research, we suggest to tighten the lower bounds in [9, Theorem 4] by relying on the approach used 
to prove the improved lower bound on the parity-check density stated in Theorem 4.1 (as compared 
to the derivation of the bound in [9, Theorem 4] which relies on the analysis in [13, Theorem 2.1]). 

Appendix A 

A.l Proof of Lemma 3.1 

(i) 

In order to prove Lemma 3.1, let P^ designate the bit error probability of the i-th symbol in the 

nR 

code C (where 1 < i < n). Therefore, P^ = ^ P^j is the average bit error probability of the 

i=i 

code, and 

n 

^ H(xi \ Y, xi, x 2 , ■ ■ ■ , Xi-i) 

i=l 

n 

nR n 

^#(xj|Y) ^2 H(x i \x 1 ,x 2 ,...,x nR ) 

i=l |_ i=nR+l 

n n 

nR 

Y,H{xi\Y) 

j=i 

n 

nR 

i=i 

n 

where equality (a) follows from the chain rule of the entropy, inequality (b) is due to the fact that 
conditioning reduces the entropy, equality (c) follows since the dimension of the code is nR which 
implies that there is a set of nR bits of the codeword whose knowledge reveals the entire codeword 
(w.l.g., one can assume that these are the first nR bits), inequality (d) is based on Fano's inequality 
and since the code is binary, and inequality (e) is based on Jensen's inequality and the concavity 
of the binary entropy function. 



H(X\Y) (a) 



n 



(b) 
< 



(c) 



(d) 
< 



(e) 
< 



A. 2 Derivation of the Optimization Equation in (22) and Proving the Existence 
of its Solution 



Derivation of the optimization equation (22): We derive here the optimization equation (22) which 
refers to the "four-level quantization" lower bound on the parity-check density (see p. 10). 

Let a(-) designate the conditional pdf of the LLR at the output of the original MBIOS channel, 
given the zero symbol is transmitted. In the following, we express the transition probabilities of 
the degraded channel in Fig. 1 (see p. 6) in terms of a(-) and the value of /: 



(u) du 



/oo 
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A 1 ,0+ 

pi = Pr(Z = a | X = 0) = / a{u) du+ - a(u) du 
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a(u) du + - a(u) du 
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p 3 = Pr(Z = 1\X = 0)= I a(i 

J —oo 



(u) du. 



(A.l) 
(A.2) 
(A.3) 
(A.4) 



We note that the integration of a(-) from u = 0~ to u = + is meaningful if and only if there is a 
non- vanishing probability that the value of the LLR at the output of the original channel is zero 
(e.g., a BEC). Otherwise, the contribution of this integral to (A.2) and (A.3) vanishes. Since the 
channel is MBIOS, the symmetry property [10] gives 



a(u) = e u a(—u), Vm€ 



(A.5) 



Based on the expressions for the coefficients K\ and K<i in the lower bound on the asymptotic 
parity-check density (20), then in order to find the tightest lower bound then we need to maximize 



(pi -P2) 2 (po -P3) 2 



Pi +P2 P0+ P3 

w.r.t. the free parameter I € M + . From Eqs. (A.1)-(A.4) and the symmetry property in (A.5) 
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t. I gives 
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Since the first derivative of a function changes its sign at a neighborhood of any local maxima or 
minima point, and since a(-) is always non-negative, then the second multiplicative term above is 



the one which changes its sign at a neighborhood of I maximizing (A. 6). For this value of I, the 
second multiplicative term vanishes, which gives the optimization equation for I in (22). 

Proof of existence of a solution to (22) : In order to show that a solution to (22) always exists, 
we will see how the LHS and the RHS of this equation behave as I — > + and / — ► oo. From 
(A.1)-(A.4), it follows that in the limit where I — > oo, we get 

pi -> 1 - w -Pr(LLR(Y) = oo | X = 0), p 2 -» io 

where u? is introduced in (2), and therefore 

p\ + e~'pf / w 
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Since from the symmetry property 
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then the fraction — tends to zero as I — > oo, so 



lim P } + e ~ l j = lim fe) + ^ = 0. (A.12) 
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It therefore follows from (A.ll) and (A.12) that for large enough values of I, the LHS of (22) is 
larger than the RHS of this equation. On the other hand, in the limit where I — > + , we get 
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and therefore 
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In the limit where / —> + 



Po 
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where (3 = 1— a{u)du. By denoting u = a(u)du, we get < n < /?, and 
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We note that the last inequality holds in equality if and only if u = ^. But if this condition holds, 
then this implies that 

r0~ roo 
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-00 Jo+ 



which from the symmetry property cannot be satisfied unless a(u) = 6(u). The latter condition 
corresponds to a BEC with erasure probability 1 (whose capacity is equal to zero). 

From (A.13) and (A.14), we obtain that for small enough (and non-negative) values of I, the 
LHS of (22) is less or equal to the RHS of this equation. Since we also obtained that for large 
enough I, the LHS of (22) is larger than the RHS of this equation, the existence of a solution to 
(22) follows from continuity considerations. 



A.3 Proof of Inequality (30) 



We prove here the inequality (30) (see p. 12) which implies that the "four-level quantization" lower 
bound on the parity-check density (see p. 10) is tighter than what can be interpreted as the "two 
levels quantization" bound in [13, Theorem 2.1]. Based on (2), we get 

w = Pr{LLR(Y) < | X = 0} + - Pr{LLR(Y) = | X = 0} 

so from (&), w = p2 + P3- By invoking Jensen's inequality, we get 
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An equality is achieved if and only if = . From (A.7)-(A.10), we get 
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The two fractions Pl , P2 and Po . p:i cannot be equal unless the LLR is either equal to I or — I. This 
makes the four-level quantization of the LLR identical to the two-level quantization used for the 
derivation of the original bound in [1, Theorem 2]. Equality can be also achieved if p± + p 2 = or 
Po +P3 = which converts the channel model in Fig. 1 (see p. 6) to a BSC. 



Appendix B 



B.l Proof of Lemma 3.3 



Lemma 3.3 (see p. 16) is proved here by mathematical induction on the value of k. 

k = 1: let s denote the value of s for which k s = 1. In this case, we simply need to find the 

probability that the scalars 6 and X differ. From (35) 
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Let us assume that for every k <k! the claim holds, and prove it for k = k'. Let s denote the value 
of s for which ^ = (a^\a^\ . . . , ajj-i)- The probability that the components of the two random 
vectors (0^ , Oj 2 , . . . , 0j fc , ) and (X^ , X,- l2 , . . . , Xj , ) differ in an even number of indices is equal to 

Ql (s) q 2 (k', 8) + (1 - qi{~s)) (1 - q 2 (k', 3)) 

where qi(s) designates the probability that 0^ = X^, and q 2 (k', s) designates the probability that 
the components of the two random vectors (0j 2 , . . . , 0j fe , ) and (Xj 2 , . . . , Xj , ) differ in an even 
number of indices. Based on the assumption, we get 
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so the probability that (0^ , . . . , 0j fc , ) and (Xj x , . . . , Xj , ) differ in an even number of indices is 

2 d-l_! 
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This completes the proof of Lemma 3.3. 



B.2 Proof of the Property Claimed in the Discussion on Proposition 3.2 



Following the discussion on Proposition 3.2 (see p. 17), we prove the existence of sub-optimal 2 d+1 
quantization levels, determined by l±, . . . ,l 2 d ~i an d their symmetric values around zero, so that 
even with this sub-optimal 2 rf+1 -level quantization, the bound in the RHS of (34) is already tighter 
than the one which follows from the optimal choice of 2 d quantization levels. From the RHS of 
(34), it suffices to show that for any integer k > 2 
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where po,Pi, ■ ■ ■ ,p 2 d+i_i denote the transition probabilities, as defined in (33), which are associated 
with the above sub-optimal 2 d+1 quantization levels. On the other hand, Pq\p"i \ ■ ■ ■ jP^-i denote 
the transition probabilities in (33) which correspond to the optimal 2 d quantization levels. 

To prove (B.l), we define sub-optimal quantization levels Zi, . . . , l 2 d_\ in the following way: For 

i = 1,2,..., 2 d ~ 1 — 1, we define In = l^f \ where lf \ . . . , ^d-i_ 1 (and their symmetric values around 

zero) are the optimal 2 d quantization levels. The other levels (i.e., lj where the index j is odd) are 
chosen arbitrarily as long as 

oo = lo > l± > . . . > l 2 d_i > 0. 

In a similar way to (33), let us the denote by po, . . . ,j5 2 d+i_ 1 the transition probabilities associated 
with Zi, l 2 , . . . , l 2 d_i- This yields 
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Let us denote 



pi +p 2 d+i_ 1 _ 
^ = k 2i + k 2i+ i, i = 0,1,..., 2^-1 
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then, the above sum transforms to 
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Due to the choice of lj, we have that = p 2 j + ft 2 j+i for any j = 0, 1, . . . , 2 d — 1. Hence, 



2<*-l_i 

e n 

Vj: fc 2:) '+A:2j+l=fcj- 
2 d -!-l 

' 



fe' \ (P2j +P2d+i-l-2i) fc2j fei+l +P 2 d 



k 2j 



-l-(2j+l), 



(B.3) 



-17 V ( k ' j \ — + ^ 2d+1 - 1 - 2 J^ 2J fe^+ 1 +^+l-l-(2j+l)) fc2j ' 



3=0 k 2j ,k 2j+ i 

k 2 j+k 2 j+i=k' j 



n 



^T 1 ((P2j +P2j+l) + fe*+l-l-2j +P2<H-i-l-(2j+l))) ' 



2 d - 



where the factorization, in (a), of the sum over the global function (whose variables are ko,k\, . . . , k 2 d_i) 
into a product of sums of local functions (with variables k 2 j, k 2 j+i) follows from the concept of factor 
graphs. Therefore, the expression inside the sum in (B.3) forms a probability distribution. 



Using the concavity of h,2(-), we apply Jensen's inequality to (B.2) (which is equal to the RHS 
of (B.l)) and get 
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which therefore proves the sufficient condition for monotonicity, as stated in (B.l). 



Appendix C 



We provide in this appendix further mathematical details related to the proof of Proposition 4.1. 
We note that Appendix C.2 serves here as a preparatory step for the derivation in Appendix C.3. 



C.l Proof of Lemma 4.1 



In order to prove Lemma 4.1, we first observe that 
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The proof of the lemma continues by mathematical induction. For k = 1, (56) holds since 
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Let us assume that (56) holds for k' < k, and prove it also for k. By our assumption and since the 
channel is memory less, we get 
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where equality (a) follows from the assumption that equation (56) holds for k — 1, and the proof 
follows by mathematical induction. 



C.2 Power Series Expansion of the Binary Entropy Function 
Lemma C.l. 

fr 2 ( x ) = i_ ' y ii o<x<i. (c.i) 

v 7 2 In 2 ^ p(2p- 1 ' - - v ^ 

p=i 

Proof. We prove this by expanding the binary entropy function into a power series around \. The 
first order derivative is 

*<») = 4^ 

and the higher order derivatives get the form 

2 v 7 In 2 V ^ (1 ~x) n - 1 J 



The derivatives of odd degree therefore vanish at x = \ , and for an even value of n > 2 
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This yields the following power series expansion of /12O) around x - -■ 
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and this power series converges for all ie [0,1]. □ 



We note that since the power series in (C.l) has always non-negative coefficients, then its 
truncation always gives an upper bound on the binary entropy function, i.e., 

"^'^-^E^V V, 6 [0,l].m 6 N. (C.2) 

The case where m = 1 gives the upper bound in Lemma 3.2 which is used in this paper for 
the derivation of the lower bounds on the parity-check density The reason for not using a tighter 
version of the binary entropy function for this case was because otherwise we would get a polynomial 
equation for cxr whose solution cannot be given necessarily in closed form. As shown in Fig. 5, 
the upper bound on the binary entropy function, ^(O) over the whole interval [0, 1] is improved 
considerably by taking even a moderate value for m (e.g., m = 10 gives already a very tight upper 
bound on foO) which deviates from the exact values only at a small neighborhood near the two 
endpoints of this interval) . 



C.3 Calculation of the Multi-Dimensional Integral in (57) 

Based on Lemma C.l which provides a power series expansion of ^(O near |, we obtain 
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2 In 

This transforms the original fc-dimensional integral to an infinite sum of one-dimensional integrals. 
Since we are interested to obtain a tight upper bound on the fc-dimensional integral above, and 
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Figure 5: Plot of the binary entropy function to base 2 and some upper bounds which are obtained 
by truncating its power series around x = |. 

since all the terms of the last infinite series are positive, then any truncation of the last infinite 
series is an upper bound. Based on the discussion in Appendix C.2, we will compute the first 10 
terms of this series which (based on the plot in Fig. 5) will give a very tight upper bound on the 
fc-dimensional integral (for all k). Hence, for the calculation of the un-quantized bounds on the 
thresholds of LDPC ensembles, we rely in our computations on the following tight upper bound on 
the /c-dimensional integral for any integer k: 

r-oo noo r-co k ( 1 ( -tt / 1 — e _Qm \ \ \ 

Jo Jo "h n / - (Qm) * 2 U ( 1 " n ( i +<,-<■- J J J ■ ■ ■ *** 

This bound is clearly much tighter than the one we use in Section 4 for the derivation of a lower 
bound on the parity-check density, as for the derivation of the latter bound, we only take into 
account the first term (among the 10 positive terms) of the sum above. 

Appendix D 

D.l Proof of Inequality (61) 

Let d > 2 be an arbitrary integer and let oo = Iq > l± > . . . > i 2 rf-i-i > (and their symmetric 
values around zero) be arbitrarily chosen quantization levels. To prove inequality (61), we start 
by applying the power series expansion of the binary entropy function (C.l) to the RHS of the 



inequality and get 
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Therefore, a sufficient condition for (61) to hold, is that for any integers k > 2 and p > 1 
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(D.l) 



Let /q(-) be the pdf oi the absolute value of the LLR, as defined in (51) (it is independent of the 
transmitted symbol X because of the symmetry of the channel). Dividing the range of integration 
[0, oo] in the LHS of (D.l) into the 2 d ~ 1 — 1 sub- intervals defined by the non- negative quantization 
levels h, ■ ■ ■ , ^d-i-i yields 
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where (a) holds since tanh(0) = 0. The above chain of equalities implies that in order to prove 
(D.l), it suffices to show that for any integer p > 1 
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From (33), we have that 
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which implies that for every i = I, . . . ,2 d 1 — 1, the function Jl 1 — is a pdf over the respective 



where (a) follows from the convexity of the function f(x) = x 2p and by applying Jensen's equality, 
(b) follows from the definition of fn(-) in (51), (c) follows from the symmetry property of the pdf 
a(-), and (d) follows from (33). For i = 0, the proof follows along the same lines as (D.3), except 
that in (b) we also use the fact that tanh(0) = 0, so the two integrals from + to Z 2 d-i-i an d from 
to l 2 d-i_ 1 get the same value. This concludes the proof of the sufficient condition (D.2) for the 
satisfiability of inequality (61). 

D.2 Proof of the Claim Regarding the Tightness of the Lower Bound on the 
Parity-Check Density in Theorem 4.1 

We show here that the lower bound on the parity-check density in Theorem 4.1 is uniformly tighter 
than the one in [13, Theorem 2.1] (except for the BSC and BEC where they coincide). In order to 
show this , we first prove the following lemma: 

Lemma D.l. For any MBIOS channel, the following inequality holds 

A > (1 - 2wf 

where w and A are introduced in (2) and (64), respectively. 



interval. Therefore, for % € {l, . . . , 2 d 1 — l}, we have 
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Proof. From (2), (51) and (64) 
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where the single inequality above follows from Jensen's inequality. 



□ 



The proof of our claim now follows directly by replacing the supremum over x G (0, A], which 
appears in the RHS of (62), with the same expression where we substitute x = (1 — 2w) 2 . 



D.3 Proof for the Claim in Remark 4.4 

In order to prove the claim in Remark 4.4 (see p. 27), it is required to show that 
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where tt; is introduced in (2). The reason for showing this in light of the claim in Remark 4.4 is 
that the RHS of the last inequality follows from considerations related to a BEC, essentially in the 
same way that the second term of the maximization in the RHS of (49) is derived. By showing 



this, we prove that the maximization of the two expressions in the LHS and RHS of (D.4) doesn't 
affect the bound in Corollary 4.1. 



Following the steps which lead to (68), we get that for any integer k > 2 
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Applying this to (D.4) and denoting Q(x) = J2k^kX h , we get that a sufficient condition for (D.4) 
to hold is 

l-fi(C) - 1-0(1 -2w)' 1 j 

From the erasure decomposition lemma, we get that an MBIOS channel is physically degraded as 
compared to a BEC with an erasure probability p = 2w. By the information processing inequality, 
it follows that C < 1 — 2w. Therefore, in order to prove (D.5), it is enough to show that the function 

f(x) = 



1 - fi(x) 

is monotonically decreasing for x € (0, 1). We prove this property by showing that the derivative 
of /(•) is non-positive for x £ (0,1). As the denominator of the derivative is positive, we may 
equivalently show 

n'(x) (1 - x) - (1 - Q(x)) <0. 

Dividing both sides of the inequality by (1 — x) G (0, 1) and noting that 0,(1) = J2k dk = 1) we get 
that it is enough to show 

a , [x) _ nm -m £ (D 6) 

Since Q(-) is a polynomial and therefore analytic, by the mean-value theorem we get that for some 
x <G (x, 1) 

1 — x 

Since Q'(x) = kd^x k is monotonically increasing for x > 0, then (D.6) follows for all x G (0, 1). 
This in turn proves (D.4). 

D.4 Proof of Eq. (75) 

In order to prove (75), we first multiply the two sides of (73) by R, and denote R = (1 — e)C. This 
gives that the lower bound on the bit error probability in (73) is non-positive if and only if 
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Unless the channel is noiseless, we get 
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Since -B < 1, the LHS of (D.7) is monotonically decreasing in e. We therefore deduce that the 
inequality (D.7) holds for e > eo, where eo is the solution of 

(l-C)B-e C(l-B) = 0. 

It can be readily seen that the solution of the last equation is given by eo defined in (75). 

D.5 Proof of Eq. (76) 

We will show both that there exists a unique eo that satisfies (76), and that the RHS of (74) is 
non-positive if and only if e > eo where eo is that unique solution. As in Appendix D.4, we begin 
by multiplying the two sides of (74) by R and denoting R = (1 — e)C. It follows that the bound in 
the RHS of (74) is trivial (non-positive) if and only if 
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We now show that the LHS of the last inequality is monotonically decreasing in e. Let us denote 
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By Dividing the derivative of / w.r.t. e by C, we get 
\ p=l 



c 



1 / 00 (2-(l-e)C)t 



oo (2-(l- e )C)t / , 

+ (1 - (1 - e)C) J> p a/^ 7 ^ log(a p ) ( -— — 
°° f ( ( 1 \\ {2 ~ ( f' 



e)Cf 



(2-(l-e)C)t 
- e )C 



From the symmetry property of a(-) and since tanh(x) < 1 then a p < 1, and it follows that 
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For p € N, let us denote ap~ (1_£)C = 1 — 5 P where < S p < 1. Therefore, we get from the previous 
inequality 
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where the second inequality follows from the inequality ln(l — x) > — jz^: which holds for x € (0, 1). 
This concludes the proof of the monotonicity of the LHS of (D.8). Observing that 
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where the first inequality follows since a p < 1 and since £ > 1 (t = 1 if and only if the code is 
cycle- free, otherwise t > 1.) The second equality follows from the last three equalities leading 
to (68). From the continuity of the function /(•) w.r.t. e, we conclude that the monotonicity 
property of /, as shown above, ensures a unique solution for (76). From (D.8), it also follows from 
the monotonicity and continuity properties of /(•) in terms of e G (0, 1) that the RHS of (74) is 
non-positive if and only if e > Eq where Eq is the unique solution of (76). 
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